将键值为多个列表的字典拆分为训练集和测试集Python

Question

假设我有一个包含两个键的字典，spam和ham，用于显示如下所示的spam和ham文本或电子邮件：

data = {
    'spam': [
        ['hi', "what's", 'going', 'on', 'sexy', 'thing'], 
        ['1-800', 'call', 'girls', 'if', "you're", 'lonely'], 
        ['sexy', 'girls', 'for', 'youuuuuu']], 
    'ham': [['hey', 'hey', 'I', 'got', 'your', 'message,', "I'll", 'be', 'home', 'soon!!!'], 
        ['Madden', 'MUT', 'time', 'boys']]}

我想把字典分成训练集和测试集（从80/20训练开始测试）。我希望分割是不考虑键的，所以只需将80%的训练集消息和20%的测试集消息子集。在这个小例子中，我们总共有5条消息（3条是垃圾邮件，2条是火腿）。我已经四处寻找解决方案，但还没有找到任何解决这种情况的方法

Answer 1

使用适当命名的^{}：

from sklearn.model_selection import train_test_split

data = {
    'spam': [
        ['hi', "what's", 'going', 'on', 'sexy', 'thing'],
        ['1-800', 'call', 'girls', 'if', "you're", 'lonely'],
        ['sexy', 'girls', 'for', 'youuuuuu']],
    'ham': [['hey', 'hey', 'I', 'got', 'your', 'message,', "I'll", 'be', 'home', 'soon!!!'],
            ['Madden', 'MUT', 'time', 'boys']]}

all_messages = [(words, k) for k, v in data.items() for words in v]

train, test = train_test_split(list(all_messages), test_size=0.2)

---

你可以，也可能应该使用更强大的东西，比如熊猫：

import pandas as pd
from sklearn.model_selection import train_test_split

data_dict = {
    'spam': [
        ['hi', "what's", 'going', 'on', 'sexy', 'thing'],
        ['1-800', 'call', 'girls', 'if', "you're", 'lonely'],
        ['sexy', 'girls', 'for', 'youuuuuu']],
    'ham': [['hey', 'hey', 'I', 'got', 'your', 'message,', "I'll", 'be', 'home', 'soon!!!'],
            ['Madden', 'MUT', 'time', 'boys']]}

df = pd.DataFrame(data=((k, words) for k, v in data_dict.items() for words in v))

print(df)

train, test = train_test_split(df, test_size=0.2)

print(train)
print(test)

输出：

      0                                                  1
0  spam               [hi, what's, going, on, sexy, thing]
1  spam           [1-800, call, girls, if, you're, lonely]
2  spam                       [sexy, girls, for, youuuuuu]
3   ham  [hey, hey, I, got, your, message,, I'll, be, h...
4   ham                          [Madden, MUT, time, boys]

      0                                                  1
1  spam           [1-800, call, girls, if, you're, lonely]
2  spam                       [sexy, girls, for, youuuuuu]
0  spam               [hi, what's, going, on, sexy, thing]
3   ham  [hey, hey, I, got, your, message,, I'll, be, h...

     0                          1
4  ham  [Madden, MUT, time, boys]

Answer 2

您可以将dict转换为元组列表，然后进行拆分

>>> l = [(sentence, k) for k,v in data.items() for sentence in v]
>>> random.shuffle(l)
>>> train_size = int(len(l)*0.8)
>>> train, test = l[:train_size], l[train_size:]
>>> len(train)
4
>>> len(test)
1

每个元素都是一对(sentence, label)

>>> test[0]
(['Madden', 'MUT', 'time', 'boys'], 'ham')