Python数据帧从列表中删除句子编号

Question

我在一个数据框中有一列（相当长的）文本，对于每一个文本，我想删除的句子索引列表。当我将文本拆分成句子时，句子索引由Spacy生成。请考虑以下例子：

import pandas as pd
import spacy
nlp = spacy.load('en_core_web_sm')

data = {'text': ['I am A. I am 30 years old. I live in NY.','I am B. I am 25 years old. I live in SD.','I am C. I am 30 years old. I live in TX.'], 'todel': [[1, 2], [1], [1, 2]]}

df = pd.DataFrame(data)

def get_sentences(text):
    text_clean = nlp(text)
    sentences = text_clean.sents
    sents_list = []
    for sentence in sentences:
        sents_list.append(str(sentence))
    return sents_list

df['text'] = df['text'].apply(get_sentences)

print(df)

其中给出了以下内容：

                                           text   todel
0  [I am A., I am 30 years old., I live in NY.]  [1, 2]
1   [I am B. I am 25 years old., I live in SD.]     [1]
2   [I am C. I am 30 years old., I live in TX.]  [1, 2]

知道我有一个非常大的数据集，每行有50多个句子要删除，您如何高效地删除存储在todel中的句子

我的预期产出是：

                                  text   todel
0                      [I live in NY.]  [1, 2]
1  [I am 25 years old., I live in SD.]     [1]
2                      [I live in TX.]  [1, 2]

Answer 1

试试这个：

import pandas as pd

data = {'text': ['I am A. I am 30 years old. I live in NY.','I am B. I am 25 years old. I live in SD.','I am C. I am 30 years old. I live in TX.'], 'todel': [[1, 2], [1], [1, 2]]}

df = pd.DataFrame(data)

def fun(sen, lst):
    return  ('.'.join(s for idx, s in enumerate(sen.split('.')) if idx+1 not in lst))

df['text'] = df.apply(lambda row : fun(row['text'],row['todel']), axis=1)

输出：

                                text   todel
0                      I live in NY.  [1, 2]
1   I am 25 years old. I live in SD.     [1]
2                      I live in TX.  [1, 2]

根据已编辑的问题进行编辑：

如果df['text']列出了不需要拆分的句子，您可以尝试以下方法：

data = {'text': [['I am A.', 'I am 30 years old.', 'I live in NY.'], 
                 ['I am B.', 'I am 25 years old.', 'I live in SD.'],
                 ['I am C.','I am 30 years old.',' I live in TX.']], 'todel': [[1, 2], [1], [1, 2]]}
df = pd.DataFrame(data)
#                                           text     todel
# 0   [I am A., I am 30 years old., I live in NY.]  [1, 2]
# 1   [I am B., I am 25 years old., I live in SD.]     [1]
# 2  [I am C., I am 30 years old.,  I live in TX.]  [1, 2]

def fun(sen, lst):
    return  [s for idx , s in enumerate(sen) if not idx+1 in lst]

df['text'] = df.apply(lambda row : fun(row['text'],row['todel']), axis=1)

print(df)

输出：

                                  text   todel
0                      [I live in NY.]  [1, 2]
1  [I am 25 years old., I live in SD.]     [1]
2                     [ I live in TX.]  [1, 2]