如何将字典保存为pickle

Question

我正在尝试使用Pickle把一个字典保存到文件里。保存字典的代码运行得很顺利，但当我在Python命令行中尝试从文件中取出这个字典时，出现了EOF错误：

>>> import pprint
>>> pkl_file = open('data.pkl', 'rb')
>>> data1 = pickle.load(pkl_file)
 Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "/usr/lib/python2.7/pickle.py", line 1378, in load
     return Unpickler(file).load()
     File "/usr/lib/python2.7/pickle.py", line 858, in load
      dispatch[key](self)
      File "/usr/lib/python2.7/pickle.py", line 880, in load_eof
      raise EOFError
      EOFError

我的代码如下。

这段代码统计了每个单词的出现频率和数据的日期（日期就是文件名）。然后把单词作为字典的键，把频率和日期的元组作为每个键的值。现在我想把这个字典用作我工作中另一个部分的输入：

def pathFilesList():
    source='StemmedDataset'
    retList = []
    for r,d,f in os.walk(source):
        for files in f:
            retList.append(os.path.join(r, files))
    return retList

def parsing():
    fileList = pathFilesList()
    for f in fileList:
        print "Processing file: " + str(f)
        fileWordList = []
        fileWordSet = set()
        fw=codecs.open(f,'r', encoding='utf-8')
        fLines = fw.readlines()
        for line in fLines:
            sWord = line.strip()
            fileWordList.append(sWord)
            if sWord not in fileWordSet:
                fileWordSet.add(sWord)
        for stemWord in fileWordSet:
            stemFreq = fileWordList.count(stemWord)
            if stemWord not in wordDict:
                wordDict[stemWord] = [(f[15:-4], stemFreq)]
            else:
                wordDict[stemWord].append((f[15:-4], stemFreq))
        fw.close()

if __name__ == "__main__":
    parsing()
    output = open('data.pkl', 'wb')
    pickle.dump(wordDict, output)
    output.close()

你觉得问题出在哪里呢？

Answer 1

如果你想找一个可以把大量数据字典保存到硬盘或者数据库的工具，并且能够使用序列化和编码（像是编码器和哈希表），那么你可以看看 klepto。

klepto 提供了一种字典的抽象方式，可以把数据写入数据库，包括把你的文件系统当作数据库来用（也就是说，可以把整个字典写到一个文件里，或者把每个条目写到自己的文件里）。对于大数据，我通常选择把字典表示为文件系统中的一个目录，每个条目对应一个文件。klepto 还提供了缓存算法，所以如果你使用文件系统作为字典的后端，你可以通过利用内存缓存来避免一些速度上的损失。

>>> from klepto.archives import dir_archive
>>> d = {'a':1, 'b':2, 'c':map, 'd':None}
>>> # map a dict to a filesystem directory
>>> demo = dir_archive('demo', d, serialized=True) 
>>> demo['a']
1
>>> demo['c']
<built-in function map>
>>> demo          
dir_archive('demo', {'a': 1, 'c': <built-in function map>, 'b': 2, 'd': None}, cached=True)
>>> # is set to cache to memory, so use 'dump' to dump to the filesystem 
>>> demo.dump()
>>> del demo
>>> 
>>> demo = dir_archive('demo', {}, serialized=True)
>>> demo
dir_archive('demo', {}, cached=True)
>>> # demo is empty, load from disk
>>> demo.load()
>>> demo
dir_archive('demo', {'a': 1, 'c': <built-in function map>, 'b': 2, 'd': None}, cached=True)
>>> demo['c']
<built-in function map>
>>> 

klepto 还有其他选项，比如 compression 和 memmode，可以用来定制你的数据存储方式（比如压缩级别、内存映射模式等）。使用数据库（比如 MySQL 等）作为后端也同样简单（接口完全一样），你只需设置 cached=False 就可以关闭内存缓存，这样每次读写都会直接去存档。

klepto 还允许你自定义编码，通过构建一个自定义的 keymap。

>>> from klepto.keymaps import *
>>> 
>>> s = stringmap(encoding='hex_codec')
>>> x = [1,2,'3',min]
>>> s(x)
'285b312c20322c202733272c203c6275696c742d696e2066756e6374696f6e206d696e3e5d2c29'
>>> p = picklemap(serializer='dill')
>>> p(x)
'\x80\x02]q\x00(K\x01K\x02U\x013q\x01c__builtin__\nmin\nq\x02e\x85q\x03.'
>>> sp = s+p
>>> sp(x)
'\x80\x02UT28285b312c20322c202733272c203c6275696c742d696e2066756e6374696f6e206d696e3e5d2c292c29q\x00.' 

你可以在这里获取 klepto：https://github.com/uqfoundation

Answer 2

# Added some code and comments.  To make the code more complete.
# Using collections.Counter to count words.

import os.path
import codecs
import pickle
from collections import Counter

wordDict = {}

def pathFilesList():
    source='StemmedDataset'
    retList = []
    for r, d, f in os.walk(source):
        for files in f:
            retList.append(os.path.join(r, files))
    return retList

# Starts to parse a corpus, it counts the frequency of each word and
# the date of the data (the date is the file name.) then saves words
# as keys of dictionary and the tuple of (freq,date) as values of each
# key.
def parsing():
    fileList = pathFilesList()
    for f in fileList:
        date_stamp = f[15:-4]
        print "Processing file: " + str(f)
        fileWordList = []
        fileWordSet = set()
        # One word per line, strip space. No empty lines.
        fw = codecs.open(f, mode = 'r' , encoding='utf-8')
        fileWords = Counter(w for w in fw.read().split())
        # For each unique word, count occurance and store in dict.
        for stemWord, stemFreq in fileWords.items():
            if stemWord not in wordDict:
                wordDict[stemWord] = [(date_stamp, stemFreq)]
            else:
                wordDict[stemWord].append((date_stamp, stemFreq))
        # Close file and do next.
        fw.close()


if __name__ == "__main__":
    # Parse all files and store in wordDict.
    parsing()

    output = open('data.pkl', 'wb')

    # Assume wordDict is global.
    print "Dumping wordDict of size {0}".format(len(wordDict))
    pickle.dump(wordDict, output)

    output.close()

当然可以！请把你想要翻译的内容发给我，我会帮你把它变得简单易懂。

Answer 3

因为这是Python2，所以你需要更明确地说明你的源代码是用什么编码写的。提到的PEP-0263文档详细解释了这个问题。我的建议是，你可以在unpickle.py的前两行加上以下内容：

#!/usr/bin/env python
# -*- coding: utf-8 -*-

# The rest of your code....

顺便说一下，如果你要经常处理非ASCII字符，使用Python3可能会更好。