使用Python的urllib模块发送请求时出现编码问题

Question

我正在尝试使用urllib从url获取响应，然后使用bs4提取数据，但是有一些url上我得到了这个异常，并且

from urllib.request import urlopen 

无法处理请求。给出以下错误：

编辑：错误：（回溯完成）

Traceback (most recent call last):
  File "G:/Internships/Botnostic Solutions/AllScrapers/careerz360/test_bs4.py", line 74, in <module>
    html = urlopen(url_link)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 525, in open
    response = self._open(req, data)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 543, in _open
    '_open', req)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 503, in _call_chain
    result = func(*args)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 1360, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 1317, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 1240, in _send_request
    self.putrequest(method, url, **skips)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 1107, in putrequest
    self._output(request.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode character '\xe8' in position 18: ordinal not in range(128)

代码片段：

from bs4 import BeautifulSoup
from urllib.request import urlopen
urls_list = []
with open('links_file.txt', 'r', encoding="utf-8") as linksFile:
    for l in linksFile:
        # url_parsed = urllib.parse.quote(l.strip())
        urls_list.append(l.strip())
linksFile.close()

将所有链接附加到一个列表，以便可以使用：urlib.urlopen（）

这是在此处创建问题的链接：

urls_list = ['https://www.careerz360.com/pakistan/protègè-global-android-developer-karachi-jobs-108853']

重新定义代码：

for url in urls_list:
    html = urlopen(url)
    if html.getcode() == 200:
        response = BeautifulSoup(html.read(), "lxml")
        company_name = response.select('h2#business-name')[0].text.strip()

使用了不同的编码：

我也尝试过不同的编码，比如：拉丁语-1、utf-8和iso等，但得到了相同的错误。 如果可能的话，请给我建议一个解决办法。 谢谢和问候

Answer 1

这个代码对我有用。在file.txt中，我有一个URL，https://www.careerz360.com/pakistan/protègè-global-android-developer-karachi-jobs-108853：

import requests
from bs4 import BeautifulSoup

urls_list = []
with open('file.txt', 'r') as linksFile:
    for l in linksFile:
        urls_list.append(l.strip())

print(urls_list)

soup = BeautifulSoup(requests.get(urls_list[0]).text, 'lxml')

company_name = soup.select('h2#business-name')[0].text.strip()

print(company_name)

印刷品：

['https://www.careerz360.com/pakistan/protègè-global-android-developer-karachi-jobs-108853']
Protègè Global

我使用的是python3.6.8

编辑：使用urlopen您需要quote()URL:

from urllib.request import urlopen
from urllib.parse import quote, urlparse
from bs4 import BeautifulSoup

urls_list = []
with open('file.txt', 'r') as linksFile:
    for l in linksFile:
        urls_list.append(l.strip())

print(urls_list)

p = urlparse(urls_list[0])
url = p.scheme + '://' + p.netloc + quote(p.path)

print(url)

html = urlopen(url)
soup = BeautifulSoup(html.read(), 'lxml')

company_name = soup.select('h2#business-name')[0].text.strip()

print(company_name)

印刷品：

['https://www.careerz360.com/pakistan/protègè-global-android-developer-karachi-jobs-108853']
https://www.careerz360.com/pakistan/prot%C3%A8g%C3%A8-global-android-developer-karachi-jobs-108853
Protègè Global