我在法语中使用伯特标记器,我得到了这个错误,但我似乎没有解决它。如果你有建议的话
Traceback (most recent call last):
File "training_cross_data_2.py", line 240, in <module>
training_data(f, root, testdir, dict_unc)
File "training_cross_data_2.py", line 107, in training_data
Xtrain_emb, mdlname = get_flaubert_layer(data)
File "training_cross_data_2.py", line 40, in get_flaubert_layer
tokenized = texte.apply((lambda x: flaubert_tokenizer.encode(x, add_special_tokens=True, max_length=512, truncation=True)))
File "/home/getalp/kelodjoe/anaconda3/envs/env/lib/python3.6/site-packages/pandas/core/series.py", line 3848, in apply
mapped = lib.map_infer(values, f, convert=convert_dtype)
File "pandas/_libs/lib.pyx", line 2329, in pandas._libs.lib.map_infer
File "training_cross_data_2.py", line 40, in <lambda>
tokenized = texte.apply((lambda x: flaubert_tokenizer.encode(x, add_special_tokens=True, max_length=512, truncation=True)))
File "/home/anaconda3/envs/env/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 907, in encode
**kwargs,
File "/home/anaconda3/envs/env/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 1021, in encode_plus
first_ids = get_input_ids(text)
File "/home/anaconda3/envs/env/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 1003, in get_input_ids
"Input is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers."
ValueError: Input is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers.
我环顾四周,想得到一个满意的答案,但所提出的任何建议似乎都行不通。Texte是数据帧
代码如下:
def get_flaubert_layer(texte): # teste is dataframe which I take from an excel file
language_model_dir= os.path.expanduser(args.language_model_dir)
lge_size = language_model_dir[16:-1] # modify when on jean zay 27:-1
print(lge_size)
flaubert = FlaubertModel.from_pretrained(language_model_dir)
flaubert_tokenizer = FlaubertTokenizer.from_pretrained(language_model_dir)
tokenized = texte.apply((lambda x: flaubert_tokenizer.encode(x, add_special_tokens=True, max_length=512, truncation=True)))
max_len = 0
for i in tokenized.values:
if len(i) > max_len:
max_len = len(i)
padded = np.array([i + [0] * (max_len - len(i)) for i in tokenized.values])
attention_mask = np.where(padded != 0, 1, 0)
我有另一个相同结构的文件,但它正在工作,但在这种情况下,我不知道为什么我会得到这个错误,我应该重新下载模型吗
该文件如下所示:
您可能需要更改此行:
到
这有两个好处:
encode
函数。这可能会加速标记化李>相关问题 更多 >
编程相关推荐