文件包含\u00c2\u00a0，转换为字符

Question

我有一个JSON文件，其中包含这样的文本

 .....wax, and voila!\u00c2\u00a0At the moment you can't use our ...

我的简单问题是如何将这些\u代码转换（而不是删除）为空格、撇号和e.t.c。。。？在

输入：带有.....wax, and voila!\u00c2\u00a0At the moment you can't use our ...的文本文件

输出：.....wax, and voila!(converted to the line break)At the moment you can't use our ...

Python代码

^{pr2}$

我所做的：

• 使用.json（）
• 任何组合.encode（）.decode（）和e.t.c的不同方法

编辑1

当我把这个文件上传到BigQuery时，我有-Â符号

大样本：

{
    "xxxx1": "...You don\u2019t nee...",
    "xxxx2": "...Gu\u00e9rer...",
    "xxxx3": "...boost.\u00a0Sit back an....",
    "xxxx4": "\" \u306f\u3058\u3081\u307e\u3057\u3066\"",
    "xxxx5": "\u00a0\n\u00a0",
    "xxxx6": "It was Christmas Eve babe\u2026",
    "xxxx7": "It\u2019s xxx xxx\u2026"
}

Python代码：

import json
import re
import codecs


def load():
    epos_export = r'{"xxxx1": "...You don\u2019t nee...","xxxx2": "...Gu\u00e9rer...","xxxx3": "...boost.\u00a0Sit back an....","xxxx4": "\" \u306f\u3058\u3081\u307e\u3057\u3066\"","xxxx5": "\u00a0\n\u00a0","xxxx6": "It was Christmas Eve babe\u2026","xxxx7": "It\u2019s xxx xxx\u2026"}'
    x = json.loads(re.sub(r"(?i)(?:\\u00[0-9a-f]{2})+", unmangle_utf8, epos_export))

    with open("TEST.json", "w") as file:
        json.dump(x,file)

def unmangle_utf8(match):
    escaped = match.group(0)                   # '\\u00e2\\u0082\\u00ac'
    hexstr = escaped.replace(r'\u00', '')      # 'e282ac'
    buffer = codecs.decode(hexstr, "hex")      # b'\xe2\x82\xac'

    try:
        return buffer.decode('utf8')           # '€'
    except UnicodeDecodeError:
        print("Could not decode buffer: %s" % buffer)



if __name__ == '__main__':
    load()

Answer 1

当您试图在一个名为TEST.json的文件中写入此内容时，我将假定此字符串是较大json字符串的一部分。在

让我举一个完整的例子：

js = '''{"a": "and voila!\\u00c2\\u00a0At the moment you can't use our"}'''
print(js)

{"a": "and voila!\u00c2\u00a0At the moment you can't use our"}

我首先用json加载它：

^{pr2}$

好了，这看起来像是一个utf-8字符串，它被错误地解码为Latin1。让我们反向操作：

x['a'] = x['a'].encode('latin1').decode('utf8')
print(x)
print(x['a'])

{'a': "and voila!\xa0At the moment you can't use our"}
and voila! At the moment you can't use our

好了，现在可以把它转换回正确的json字符串：

print(json.dumps(x))

{"a": "and voila!\\u00a0At the moment you can\'t use our"}

表示正确编码的不间断空格（U+00A0）

你应该做的是：

# load the string as json:
js = json.loads(request)

# identify the string values in the json - you probably know how but I don't...
...

# convert the strings:
js[...] = js[...].encode('latin1').decode('utf8')

# convert back to a json string
request = json.dumps(js)