UnicodeDecodeError utf8编解码器Python 2.7

Question

我构建了一个scraper，它从csv文件读取artistnames，并通过Songkickapi从这些艺术家那里收集artistdata。但是，运行代码一段时间后，我得到以下错误：

  File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 64-65: invalid continuation byte

可以下载示例数据here：

我对编码比较陌生，我想知道如何解决这个错误？下面你可以找到我的代码。在

^{pr2}$

Traceback (most recent call last):
  File "C:\Users\rmlj\Dropbox\songkick\scrapers\Data\Scraper.py", line 45, in <module>
    load_artists()
  File "C:C:\Users\rmlj\Dropbox\songkick\scrapers\Data\Scraper.py".py", line 25, in load_artists
    r = requests.get(url)
  File "C:\Python27\lib\site-packages\requests\api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Python27\lib\site-packages\requests\api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 474, in request
    prep = self.prepare_request(req)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 407, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "C:\Python27\lib\site-packages\requests\models.py", line 302, in prepare
    self.prepare_url(url, params)
  File "C:\Python27\lib\site-packages\requests\models.py", line 358, in prepare_url
    url = url.decode('utf8')
  File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 64-65: invalid continuation byte

Answer 1

尽可能使用unicode。请求应将url中的任何非ascii字符转换为正确的编码。在

>>> import requests  

>>> requests.get(u'http://Motörhead.com/?q=Motörhead').url  
u'http://xn motrhead-p4a.com/?q=Mot%C3%B6rhead'

如您所见，域名被编码为punycode，查询字符串使用percent-encoding。在

只要artist是一个有效的unicode字符串，这就可以了。在

^{pr2}$

如果artist是字节字符串，则必须使用正确的编码将其解码为unicode，这取决于原始输入文件的编码方式。在

artist = artist.decode('SHIFT-JIS')