如何在Dataframe列中排序和填充文本

Question

我想转换一个数据集或创建一个新的数据集，该数据集以带有标签的数据集列作为输入，根据预定义的长度（如果需要，还可以使用pad）自动生成字符串序列。下面的例子应该说明我的意思。你知道吗

我能够基于ngrams手动创建一个新的数据帧。这显然在计算上非常昂贵，并且会创建许多包含重复单词的列。你知道吗

                                               text  labels
0  from dbl visual com david b lewis subject comp...       5
1  from johan blade stack urc tue nl johan wevers...      11
2  from mzhao magnus acs ohio state edu min zhao ...       6
3  from lhawkins annie wellesley edu r lee hawkin...      14
4  from seanmcd ac dal ca subject powerpc ruminat...       4

例如，将序列长度4转换为如下内容：

   text                                      labels
0  from dbl visual com                            5
1  david b lewis subject                          5
2  comp windows x frequently                      5
3  asked questions <PAD> <PAD>                    5
4  from johan blade stack                        11
5  urc tue nl johan                              11
6  wevers subject re <PAD>                       11
7  from mzhao magnus acs                          6
8  ohio state edu min                             6
9  zhao subject composite <PAD>                   6

如前所述，我能够基于ngrams创建一个新的数据帧。理论上，我可以在之后再次删除所有n行。你知道吗

    df = pd.read_csv('data.csv')
    longform = pd.DataFrame(columns=['text', 'labels'])

    for idx, content in df.iterrows():
        name_words = (i.lower() for i in content[0].split())
        ngramlis = list(ngrams(name_words,20))
        longform = longform.append(
            [{'words': ng, 'labels': content[1]} for ng in ngramlis],
            ignore_index=True
        )
    longform['text_new'] = longform['words'].apply(', '.join)
    longform['text_new'] = longform['text_new'].str.replace(',', '')

这是非常糟糕的代码，这就是为什么我很有信心，有人可能会想出更好的解决方案。你知道吗

提前谢谢！你知道吗

Answer 1

使用pandas.DataFrame.explode。你知道吗

将单词分成大小均匀的块（并填充），然后：

def chunks(l, n):
    """Yield successive n-sized chunks from l.
    Source: https://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks
    """
    n_items = len(l)
    if n_items % n:
        n_pads = n - n_items % n
    else:
        n_pads = 0
    l = l + ['<PAD>' for _ in range(n_pads)] 
    for i in range(0, len(l), n):
        yield l[i:i + n]

df['text'] = df['text'].str.split().apply(lambda x: list(chunks(x, 4)))
df = df.explode('text').reset_index(drop=True)
df['text'] = df['text'].apply(' '.join)
print(df)

输出：

                             text  labels
0             from dbl visual com       5
1           david b lewis subject       5
2          comp <PAD> <PAD> <PAD>       5
3          from johan blade stack      11
4                urc tue nl johan      11
5        wevers <PAD> <PAD> <PAD>      11
6           from mzhao magnus acs       6
7              ohio state edu min       6
8          zhao <PAD> <PAD> <PAD>       6
9   from lhawkins annie wellesley      14
10               edu r lee hawkin      14
11            from seanmcd ac dal       4
12     ca subject powerpc ruminat       4