在Linux上使用spark在HDInsight集群中尝试使用希伯来语字符UTF-8TSV文件,我得到了编码错误,有什么建议吗?在
这是我的pyspark笔记本代码:
from pyspark.sql import *
# Create an RDD from sample data
transactionsText = sc.textFile("/people.txt")
header = transactionsText.first()
# Create a schema for our data
Entry = Row('id','name','age')
# Parse the data and create a schema
transactionsParts = transactionsText.filter(lambda x:x !=header) .map(lambda l: l.encode('utf-8').split("\t"))
transactions = transactionsParts.map(lambda p: Entry(str(p[0]),str(p[1]),int(p[2])))
# Infer the schema and create a table
transactionsTable = sqlContext.createDataFrame(transactions)
# SQL can be run over DataFrames that have been registered as a table.
results = sqlContext.sql("SELECT name FROM transactionsTempTable")
# The results of SQL queries are RDDs and support all the normal RDD operations.
names = results.map(lambda p: "name: " + p.name)
for name in names.collect():
print(name)
错误:
'ascii' codec can't encode characters in position 6-11: ordinal not in range(128) Traceback (most recent call last): UnicodeEncodeError: 'ascii' codec can't encode characters in position 6-11: ordinal not in range(128)
希伯来文文本文件内容:
^{pr2}$当我尝试使用英语文件时,效果很好:
英文文本文件内容:
id name age
1 guy 37
2 maor 32
3 danny 55
输出:
name: guy
name: maor
name: danny
如果使用希伯来语文本运行以下代码:
您会注意到,这些名称是
^{pr2}$unicode
类型的列表:现在,我们用事务RDD注册一个表:
您会注意到,从
sqlContext.sql(...
返回的PysparkDataFrame
中的所有字符串都是Pythonunicode
类型:正在运行:
将获得预期结果:
请注意,如果您想对这些名称进行一些处理,您应该将它们作为
unicode
字符串来处理。来自this article:相关问题 更多 >
编程相关推荐