如果python中有一个庞大的字典,如何减少内存并提高速度?

2024-09-30 01:36:52 发布

您现在位置:Python中文网/ 问答频道 /正文

我想查一本字典,它的值(字符串)有特殊的单词和关键字。字典已将加载转储到多个文件中(其中任何文件都有几百个maga字节,因此它们的总和按GBs的顺序排列)。所以速度太慢了,你有没有更好的方法来加速程序?我的代码如下:

teachertag=['K', 'curriculum', 'School', 'childhood', ]

import pickle
import re
import os
import time
start_time = time.time()


itemlist2=[]
for i in document:
    with open (i, 'rb') as fp:
        itemlist = pickle.load(fp)
        itemlist2+=itemlist



teacher=[]
parentandstudent=[]
parentandstudenttweet=[]
teachertweet=[]
allpeople=[]
i=0
for user in itemlist2:    
    if user.lang == 'en' and user.description!='':
        allpeople+=[user.screen_name]
        wordList = re.sub("[^\w]", " ",  user.description).split()
        k=0
        for j in wordList:
            k+=1
            if j in teachertag:
                teacher+=[user.screen_name]
                teachertweet+=[user.description]

                break
        else:
            parentandstudent+=[user.screen_name]
            parentandstudenttweet+=[user.description]

Tags: 文件nameinimportrefor字典time

热门问题