pyspark中的解析地址函数

Question

我想在PySpark中的数据帧中添加一列，地址通过libpostal库解析

import pyspark.sql.functions as sf
from postal.parser import parse_address
df = spark.read.parquet(path_hdfs)
df = df.select("id", "name" ,"street", "cty")\
       .withColumn("address", parse_address(sf.concat(col("street"),sf.lit(" ") ,col("cty"))))\
       .dropDuplicates()

但我得到了一个错误：

TypeError: cannot convert 'Column' object to bytes

我如何更改parse_address的参数才能被该函数接受

Answer 1

您需要使用UDF来调用Python库，例如

df = df.select("id", "name" ,"street", "cty")\
       .withColumn("address", 
           sf.udf(parse_address, 'array<struct<val:string,key:string>>')
           (sf.concat("street", sf.lit(" "), "cty"))
       )\
       .dropDuplicates()