TypeError:从HTML进行分析时，出现“Request”对象不可编辑的错误

Question

我用beauthoulsoup编写了一个脚本来解析来自一个网站的信息，但是我对它有问题。在

从代码中可以看出，在websiteReq的注释部分，它给出的错误是

引发HTTPError(请求完整网址，代码、消息、hdrs、fp） HTTPError:禁止

我在网上搜索过，并推荐使用请求。正如代码中所示，我尝试了它，但得到的错误是TypeError:'Request'object is not iterable。在

我想知道我哪里做错了。谢谢您。祝您有个美好的一天。在

代码：

   # -*- coding: utf-8 -*-
"""
Created on Tue Jul 23 22:39:05 2019

@author: Nijat
"""

try:
    # For Python 3.0 and later
    from urllib.request import Request, urlopen
except ImportError:
    # Fall back to Python 2's urllib2
    from urllib2 import urlopen

from bs4 import BeautifulSoup
import csv 
from datetime import datetime
import requests

webData = []

#websiteReq = ['https://boss.az/vacancies'] #FORBIDDEN ERROR WITH THIS 

websiteReq = Request('https://boss.az/vacancies', headers={'User-Agent': 'Mozilla/5.0'}) #Request not iterable

#for pg in websiteReq: #ERROR OCCURS HERE
page = urlopen(websiteReq)
soupping = BeautifulSoup(page, 'html.parser').encode("utf-8")

takingTitle = soupping.find_all('h3', attrs={'class':'results-i-title'})
takingCompany = soupping.find_all('a', attrs={'class':'results-i-company'})
takingDescription = soupping.find_all('div', attrs={'class':'results-i-summary'})

nameofVac = takingTitle.text.strip()
nameofComp = takingCompany.text.strip()
nameofDescript = takingDescription.text.strip()

webData.append((nameofVac, nameofComp, nameofDescript))

with open('Vacancies.csv','a') as csvfile:
    writingtocsv = csv.writer(csvfile)
    for nameofVac, nameofComp in webData:
        writingtocsv.writerow([nameofVac, nameofComp, nameofDescript,datetime.now()])

来自应答的更新代码：

所以，结果给了我从第一页分析的所有信息，大约有30个空缺。如果您想添加其他页面，只需执行+=

^{pr2}$

但是，它现在读不到所有的页面。有什么线索吗？在

Answer 1

Request（）返回类Request的对象。这是不可忍受的。您应该使用这个类中的一些方法来返回文本/内容。然后你可以用BS来解析它。在

>>> import requests
>>> response = requests.get('https://httpbin.org/get')
>>> print(response.content)
b'{\n  "args": {}, \n  "headers": {\n    "Accept": "*/*", \n    "Accept-Encoding": "gzip, deflate", \n    "Host": "httpbin.org", \n    "User-Agent": "python-requests/2.9.1"\n  }, \n  "origin": "95.56.82.136", \n  "url": "https://httpbin.org/get"\n}\n'
>>> response.json()
{'headers': {'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'python-requests/2.9.1', 'Host': 'httpbin.org', 'Accept': '*/*'}, 'args': {}, 'origin': '95.56.82.136', 'url': 'https://httpbin.org/get'}
>>> response.headers
{'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Server': 'nginx', 'Access-Control-Allow-Credentials': 'true', 'Access-Control-Allow-Origin': '*', 'Content-Length': '237', 'Date': 'Wed, 23 Dec 2015 17:56:46 GMT'}
>>> response.headers.get('Server')
'nginx'