如何正确保存文件避免UnicodeDecodeE

Question

我有一个很大的数据集，在尝试使用数据集时会抛出一个UnicodeDecodeError。你知道吗

我有一个big_file_format_x并希望将其格式化为big_file_format_y，如下所示：

with open(PATH, "r") as data:
    for index, line in enumerate(data.readlines()):
        // Formatting logic

with open(SAVE_PATH, "w") as new_data:
    new_data.write(formated_data_string)

然后，我将格式化的数据集拆分为3个数据集：

with open(PATH, "rb") as data:
    for line in data.readlines():
        // Do splitting logic

with open('train_data.txt', 'wb') as file:
    file.write(train_data)

with open('vali_data.txt', 'wb') as file:
    file.write(vali_data)

with open('test_data.txt', 'wb') as file:
    file.write(test_data)

现在当我想处理我得到的数据集时

    kwargs[name] = annotation.from_params(params=subparams, **subextras)
  File "/home/public/s1234/.conda/envs/allennlp/lib/python3.6/site-packages/allennlp/common/from_params.py", line 274, in from_params
    return subclass.from_params(params=params, **extras)
  File "/home/public/s1234/.conda/envs/allennlp/lib/python3.6/site-packages/allennlp/modules/text_field_embedders/basic_text_field_embedder.py", line 136, in from_params
    for name, subparams in token_embedder_params.items()
  File "/home/public/s1234/.conda/envs/allennlp/lib/python3.6/site-packages/allennlp/modules/text_field_embedders/basic_text_field_embedder.py", line 136, in <dictcomp>
    for name, subparams in token_embedder_params.items()
  File "/home/public/s1234/.conda/envs/allennlp/lib/python3.6/site-packages/allennlp/common/from_params.py", line 274, in from_params
    return subclass.from_params(params=params, **extras)
  File "/home/public/s1234/.conda/envs/allennlp/lib/python3.6/site-packages/allennlp/modules/token_embedders/embedding.py", line 200, in from_params
    vocab_namespace)
  File "/home/public/s1234/.conda/envs/allennlp/lib/python3.6/site-packages/allennlp/modules/token_embedders/embedding.py", line 270, in _read_pretrained_embeddings_file
    vocab, namespace)
  File "/home/public/s1234/.conda/envs/allennlp/lib/python3.6/site-packages/allennlp/modules/token_embedders/embedding.py", line 293, in _read_embeddings_from_text_file
    with EmbeddingsTextFile(file_uri) as embeddings_file:
  File "/home/public/s1234/.conda/envs/allennlp/lib/python3.6/site-packages/allennlp/modules/token_embedders/embedding.py", line 450, in __init__
    first_line = next(self._handle)     # this moves the iterator forward
  File "/home/public/s1234/.conda/envs/allennlp/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 0: invalid start byte

如何避免这个错误？你知道吗

Answer 1

定义编码应该解决这个问题：

open(PATH, "r", encoding="UTF-8") # replace UTF-8 with the encoding of your file

保存文件时，还应定义编码。否则将使用默认编码：python 3.0 open() default encoding

Answer 2

该文件中的编码可能与utf-8不同。0xba是ISO-8869-1编码中的字符。你知道吗

试试看

data.encode ("utf-8")