Pyspark或sparksql解决方案，用于剥离字符串列中第三个单词之后的所有内容

Question

我有一个带有如下文本列的数据框：

product 
"HI Celebrate Cake White    612612" 
"GO Choc Celebrat bl    682222" 
"BI Chocolate Buttercream f34   9282222" 
"Graduation Cake    28" 
"Slab Image Cake  13273" 
"Slab Celebration Cake  2556659" 
"Grain Bread" 

我想在第三个字之后把所有的东西都去掉。如何使用pyspark或sparksql来执行此操作？你知道吗

Answer 1

我找到了解决办法：

from pyspark.sql.functions import regexp_extract, col, split
from pyspark.sql import functions as sf 
df_test=spark.sql("select * from brand_cleanup")
#Applying the transformations to the data

split_col=split(df_test.item_eng_desc,' ')
df_split=df_test.withColumn('item_desc_clean',sf.concat(split_col.getItem(0),sf.lit(' '),split_col.getItem(1),sf.lit(' '),split_col.getItem(2)))

Answer 2

您可以使用正则表达式来提取前三个单词。你知道吗

df.select(regexp_extract(col("product"), "([^\\s]+\\s+){0,2}[^\\s]+", 0))\
  .show(truncate=False)

+                         +
|regexp_extract(product, ([^\s]+\s+){0,2}[^\s]+, 0)|
+                         +
|HI Celebrate Cake                                 |
|GO Choc Celebrat                                  |
|BI Chocolate Buttercream                          |
|Graduation Cake    28                             |
|Slab Image Cake                                   |
|Slab Celebration Cake                             |
|Grain Bread                                       |
+                         +