在python中撤消标记化

Question

我想逆转我应用于数据的标记化

data = [['this', 'is', 'a', 'sentence'], ['this', 'is', 'a', 'sentence', '2']]

预期产出：

['this is a sentence', 'this is a sentence 2']

我尝试使用以下代码块执行此操作：

from nltk.tokenize.treebank import TreebankWordDetokenizer
data_untoken= []
for i, text in enumerate(data):
    data_untoken.append(text)
    data_untoken = TreebankWordDetokenizer().detokenize(text)

但是我有以下错误

'str' object has no attribute 'append'

Answer 1

使用join()：

def untokenize(data):
    for tokens in data:
        yield ' '.join(tokens)


data = [['this', 'is', 'a', 'sentence'], ['this', 'is', 'a', 'sentence', '2']]
untokenized_data = list(untokenize(data))