我正在尝试使用WordLevel
/BPE
标记器对一些数字字符串进行标记,创建一个数据压缩程序,并最终在PyTorch数据加载器中使用它来从头训练一个新模型
然而,我得到了一个错误
AttributeError: 'ByteLevelBPETokenizer' object has no attribute 'pad_token_id'
当运行以下代码时
from transformers import DataCollatorForLanguageModeling
from tokenizers import ByteLevelBPETokenizer
from tokenizers.pre_tokenizers import Whitespace
from torch.utils.data import DataLoader, TensorDataset
data = ['4814 4832 4761 4523 4999 4860 4699 5024 4788 <unk>']
# Tokenizer
tokenizer = ByteLevelBPETokenizer()
tokenizer.pre_tokenizer = Whitespace()
tokenizer.train_from_iterator(data, vocab_size=1000, min_frequency=1,
special_tokens=[
"<s>",
"</s>",
"<unk>",
"<mask>",
])
# Data Collator
data_collator = DataCollatorForLanguageModeling(
tokenizer=tokenizer, mlm=False
)
train_dataset = TensorDataset(torch.tensor(tokenizer(data, ......)))
# DataLoader
train_dataloader = DataLoader(
train_dataset,
collate_fn=data_collator
)
这个错误是因为没有为标记器配置pad_token_id
吗?如果是的话,我们怎么做
谢谢
错误跟踪:
AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/opt/anaconda3/envs/x/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
data = fetcher.fetch(index)
File "/opt/anaconda3/envs/x/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
return self.collate_fn(data)
File "/opt/anaconda3/envs/x/lib/python3.8/site-packages/transformers/data/data_collator.py", line 351, in __call__
if self.tokenizer.pad_token_id is not None:
AttributeError: 'ByteLevelBPETokenizer' object has no attribute 'pad_token_id'
康达套餐
pytorch 1.7.0 py3.8_cuda10.2.89_cudnn7.6.5_0 pytorch
pytorch-lightning 1.2.5 pyhd8ed1ab_0 conda-forge
tokenizers 0.10.1 pypi_0 pypi
transformers 4.4.2 pypi_0 pypi
该错误告诉您标记器需要一个名为
pad_token_id
的属性。您可以将ByteLevelBPETokenizer
包装到具有这样一个属性的类中(…并且遇到了其他丢失的属性),或者使用transformers库中的包装类:相关问题 更多 >
编程相关推荐