Pandas：在500万行上使用Apply和regex字符串匹配
回答此问题可获得 20 贡献值，回答如果被采纳可获得 50 分。
<p><strong>问题：</strong>我正在尝试根据<code>description</code>列对数据帧的每一行进行适当的分类。为了做到这一点，我想根据常用词的列表提取关键字。首先，我把关键短语分成词（即‘Food Store’变成‘Food’和‘Store’）。然后，我检查数据帧中是否有行同时包含单词“Food”和“Store”。不幸的是，我生成的代码太慢了。如何优化它以处理500万行数据？在</p>
<p><strong>样本数据：</strong></p>
<p>以下是我的数据帧的前30行：</p>
<pre><code> bank_report_id transaction_date amount description type_codes category
0 14698 2016-04-26 -3.00 Simply Save TD EVERY DAY SAVINGS ACCOUNT xxxxx... Savings
1 14698 2016-04-25 -110.00 ROGERSWL 1TIME _V Uncategorized
2 14698 2016-04-25 -10.50 SUBWAY # x6664 Restaurants/Dining
3 14698 2016-04-25 -1.00 Simply Save TD EVERY DAY SAVINGS ACCOUNT xxxxx... Savings
4 14698 2016-04-25 -73.75 TICKETMASTER CA Entertainment
5 14698 2016-04-25 -6.20 HAPPY ONE STOP Home Improvement
6 14698 2016-04-25 -7.74 BOOSTERJUICE-19 Restaurants/Dining
7 14698 2016-04-25 -28.49 LEISURE-FIRST O Uncategorized
8 14698 2016-04-22 -3.16 MCDONALD'S #400 Restaurants/Dining
9 14698 2016-04-22 -0.50 Simply Save TD EVERY DAY SAVINGS ACCOUNT xxxxx... Savings
10 14698 2016-04-22 -10.50 SUBWAY # x6664 Restaurants/Dining
11 14698 2016-04-21 -19.87 TRAFALGAR ESSO Gasoline/Fuel
12 14698 2016-04-21 -1.00 Simply Save TD EVERY DAY SAVINGS ACCOUNT xxxxx... Savings
13 14698 2016-04-20 -3.76 MCDONALD'S #400 Restaurants/Dining
14 14698 2016-04-20 -1.00 Simply Save TD EVERY DAY SAVINGS ACCOUNT xxxxx... Savings
15 14698 2016-04-20 -40.00 TRAFALGAR ESSO Gasoline/Fuel
16 14698 2016-04-19 -10.07 TRAFALGAR ESSO Gasoline/Fuel
17 14698 2016-04-19 -5.21 TIM HORTONS #24 Restaurants/Dining
18 14698 2016-04-19 -3.50 Simply Save TD EVERY DAY SAVINGS ACCOUNT xxxxx... Savings
19 14698 2016-04-18 -1.00 Simply Save TD EVERY DAY SAVINGS ACCOUNT xxxxx... Savings
20 14698 2016-04-18 -5.21 TIM HORTONS #24 Restaurants/Dining
21 14698 2016-04-18 -22.57 WAL-MART #3170 General Merchandise
22 14698 2016-04-18 -16.94 URBAN PLANET #1 Clothing/Shoes
23 14698 2016-04-18 -12.95 LCBO/RAO #0545 Restaurants/Dining
24 14698 2016-04-18 -13.87 TRAFALGAR ESSO Gasoline/Fuel
25 14698 2016-04-18 -41.75 NON-TD ATM W/D ATM/Cash Withdrawals
26 14698 2016-04-18 -4.19 SUBWAY # x6338 Restaurants/Dining
27 14698 2016-04-15 -0.50 Simply Save TD EVERY DAY SAVINGS ACCOUNT xxxxx... Savings
28 14698 2016-04-15 -35.06 UNION BURGER Restaurants/Dining
29 14698 2016-04-15 -25.00 PIONEER STN #1 Electronics
</code></pre>
<p>以下是单词列表的一小部分：</p>
^{pr2}$
<p><strong>我的解决方案：</strong></p>
<pre><code>def get_matches(row):
keywords = pd.read_csv('Keywords.csv', encoding='ISO-8859-1')['description'].apply(lambda x: x.lower()).str.split(
" ").tolist()
split_description = [d.lower() for d in row['description'].split(" ")]
thematches = []
for group in keywords:
matches = [any([bool(re.search(y, x)) for x in split_description]) for y in group]
if all(matches):
thematches.<a href="https://www.cnpython.com/list/append" class="inner-link">append</a>(" ".join(group))
if len(thematches) > 0:
return thematches
else:
return "NA"
df['match'] = df.apply(get_matches, axis=1)
</code></pre>
<p><strong>期望输出：</strong></p>
<pre><code> bank_report_id transaction_date amount description type_codes category match
0 14698 2016-04-26 -3.00 Simply Save TD EVERY DAY SAVINGS ACCOUNT xxxxx... Savings [simply save]
1 14698 2016-04-25 -110.00 ROGERSWL 1TIME _V Uncategorized [rogers]
2 14698 2016-04-25 -10.50 SUBWAY # x6664 Restaurants/Dining [subway]
3 14698 2016-04-25 -1.00 Simply Save TD EVERY DAY SAVINGS ACCOUNT xxxxx... Savings [simply save]
4 14698 2016-04-25 -73.75 TICKETMASTER CA Entertainment [ticket master]
5 14698 2016-04-25 -6.20 HAPPY ONE STOP Home Improvement NA
6 14698 2016-04-25 -7.74 BOOSTERJUICE-19 Restaurants/Dining [juice]
7 14698 2016-04-25 -28.49 LEISURE-FIRST O Uncategorized NA
8 14698 2016-04-22 -3.16 MCDONALD'S #400 Restaurants/Dining [mcdonald's]
9 14698 2016-04-22 -0.50 Simply Save TD EVERY DAY SAVINGS ACCOUNT xxxxx... Savings [simply save]
10 14698 2016-04-22 -10.50 SUBWAY # x6664 Restaurants/Dining [subway]
11 14698 2016-04-21 -19.87 TRAFALGAR ESSO Gasoline/Fuel [esso]
12 14698 2016-04-21 -1.00 Simply Save TD EVERY DAY SAVINGS ACCOUNT xxxxx... Savings [simply save]
13 14698 2016-04-20 -3.76 MCDONALD'S #400 Restaurants/Dining [mcdonald's]
14 14698 2016-04-20 -1.00 Simply Save TD EVERY DAY SAVINGS ACCOUNT xxxxx... Savings [simply save]
15 14698 2016-04-20 -40.00 TRAFALGAR ESSO Gasoline/Fuel [esso]
16 14698 2016-04-19 -10.07 TRAFALGAR ESSO Gasoline/Fuel [esso]
17 14698 2016-04-19 -5.21 TIM HORTONS #24 Restaurants/Dining [tim hortons, rt]
18 14698 2016-04-19 -3.50 Simply Save TD EVERY DAY SAVINGS ACCOUNT xxxxx... Savings [simply save]
19 14698 2016-04-18 -1.00 Simply Save TD EVERY DAY SAVINGS ACCOUNT xxxxx... Savings [simply save]
20 14698 2016-04-18 -5.21 TIM HORTONS #24 Restaurants/Dining [tim hortons, rt]
21 14698 2016-04-18 -22.57 WAL-MART #3170 General Merchandise [rt]
22 14698 2016-04-18 -16.94 URBAN PLANET #1 Clothing/Shoes [urban planet]
23 14698 2016-04-18 -12.95 LCBO/RAO #0545 Restaurants/Dining NA
24 14698 2016-04-18 -13.87 TRAFALGAR ESSO Gasoline/Fuel [esso]
25 14698 2016-04-18 -41.75 NON-TD ATM W/D ATM/Cash Withdrawals NA
26 14698 2016-04-18 -4.19 SUBWAY # x6338 Restaurants/Dining [subway]
27 14698 2016-04-15 -0.50 Simply Save TD EVERY DAY SAVINGS ACCOUNT xxxxx... Savings [simply save]
28 14698 2016-04-15 -35.06 UNION BURGER Restaurants/Dining [burger]
29 14698 2016-04-15 -25.00 PIONEER STN #1 Electronics [pioneer]
</code></pre>
请先 登录 后评论
针对此问题: 更多的回答
- 89 关注
- 1 收藏，216 浏览
- 网友 提问于 2天前
相关Python问题
- 如何在python中从数据帧列中删除分类值？ 9 回答
- 如何在python中从数据帧列表中删除引号 8 回答
- 如何在python中从数据帧创建列表 9 回答
- 如何在Python中从数据帧创建嵌套的JSON 6 回答
- 如何在Python中从数据帧显示wordcloud 6 回答
- 如何在Python中从数据帧的时间戳中删除字符 9 回答
- 如何在Python中从数据帧绘制简单绘图？ 6 回答
- 如何在python中从数据帧行提取具有特定长度的范围？ 1 回答
- 如何在python中从数据帧设置dict中的值 10 回答
- 如何在Python中从数据库中获得一个结果 8 回答
- 如何在python中从数据框中绘制分类条形图 10 回答
- 如何在Python中从数据框中选择特定细节？ 2 回答
- 如何在python中从数据集中删除unicode 3 回答
- 如何在python中从数据集中删除某些数值？ 8 回答
- 如何在python中从数据集中选择行 6 回答
- 如何在Python中从数组中删除元素 2 回答
- 如何在python中从数组中删除单个倒逗号？ 4 回答
- 如何在python中从数组中删除对象？ 1 回答
- 如何在python中从数组中删除引号 9 回答
- 如何在python中从数组中删除所有最小值 8 回答
