在gensim word2vec embedding tutorial之后,我训练了一个简单的word2vec模型:
from gensim.test.utils import common_texts
from gensim.models import Word2Vec
model = Word2Vec(sentences=common_texts, size=100, window=5, min_count=1, workers=4)
model.save("/content/word2vec.model")
我想想象一下using the Embedding Projector in TensorBoardThere is another straightforward tutorial in gensim documentation。我在Colab做了以下工作:
!python3 -m gensim.scripts.word2vec2tensor -i /content/word2vec.model -o /content/my_model
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.7/dist-packages/gensim/scripts/word2vec2tensor.py", line 94, in <module>
word2vec2tensor(args.input, args.output, args.binary)
File "/usr/local/lib/python3.7/dist-packages/gensim/scripts/word2vec2tensor.py", line 68, in word2vec2tensor
model = gensim.models.KeyedVectors.load_word2vec_format(word2vec_model_path, binary=binary)
File "/usr/local/lib/python3.7/dist-packages/gensim/models/keyedvectors.py", line 1438, in load_word2vec_format
limit=limit, datatype=datatype)
File "/usr/local/lib/python3.7/dist-packages/gensim/models/utils_any2vec.py", line 172, in _load_word2vec_format
header = utils.to_unicode(fin.readline(), encoding=encoding)
File "/usr/local/lib/python3.7/dist-packages/gensim/utils.py", line 355, in any2unicode
return unicode(text, encoding, errors=errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
请注意,我确实首先检查了这个exact same question from 2018,但接受的答案不再有效,因为gensim和tensorflow中的答案都已更新,因此我认为值得在2021年第4季度再次询问
以原始C word2vec实现格式保存模型可以解决以下问题:
model.wv.save_word2vec_format("/content/word2vec.model")
:在
gensim
中存储word2vec模型有两种格式:来自原始word2vec实现的键控向量格式和另外存储隐藏权重、词汇频率等的格式。示例和详细信息可在documentation中找到。脚本word2vec2tensor.py
使用原始格式并使用load_word2vec_format
:code加载模型相关问题 更多 >
编程相关推荐