我试图加载两个数据集,并将它们用于培训
包版本:python 3.7; pytorch 1.3.1
可以单独创建数据加载程序,并按顺序对其进行培训:
from torch.utils.data import DataLoader, ConcatDataset
train_loader_modelnet = DataLoader(ModelNet(args.modelnet_root, categories=args.modelnet_categories,split='train', transform=transform_modelnet, device=args.device),batch_size=args.batch_size, shuffle=True)
train_loader_mydata = DataLoader(MyDataset(args.customdata_root, categories=args.mydata_categories, split='train', device=args.device),batch_size=args.batch_size, shuffle=True)
for e in range(args.epochs):
for idx, batch in enumerate(tqdm(train_loader_modelnet)):
# training on dataset1
for idx, batch in enumerate(tqdm(train_loader_custom)):
# training on dataset2
注意:MyDataset是一个自定义的dataset类,它已经实现了def __len__(self):
{
但理想情况下,我希望将它们组合成一个dataloader对象。我根据pytorch文档进行了尝试:
train_modelnet = ModelNet(args.modelnet_root, categories=args.modelnet_categories,
split='train', transform=transform_modelnet, device=args.device)
train_mydata = CloudDataset(args.customdata_root, categories=args.mydata_categories,
split='train', device=args.device)
train_loader = torch.utils.data.ConcatDataset(train_modelnet, train_customdata)
for e in range(args.epochs):
for idx, batch in enumerate(tqdm(train_loader)):
# training on combined
但是,在随机批处理中,我得到了以下“参数0中的元素X应为张量,但得到了元组”类型的错误。任何帮助都将不胜感激
> 40%|████ | 53/131 [01:03<02:00, 1.55s/it]
> Traceback (mostrecent call last): File
> "/home/chris/Programs/pycharm-anaconda-2019.3.4/plugins/python/helpers/pydev/pydevd.py",
> line 1434, in _exec
> pydev_imports.execfile(file, globals, locals) # execute the script File
> "/home/chris/Programs/pycharm-anaconda-2019.3.4/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
> exec(compile(contents+"\n", file, 'exec'), glob, loc) File "/home/chris/Documents/4yp/Data/my_kaolin/Classification/pointcloud_classification_combinedset.py",
> line 83, in <module>
> for idx, batch in enumerate(tqdm(train_loader)): File "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/tqdm/std.py",
> line 1107, in __iter__
> for obj in iterable: File "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/torch/utils/data/dataloader.py",
> line 346, in __next__
> data = self._dataset_fetcher.fetch(index) # may raise StopIteration File
> "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py",
> line 47, in fetch
> return self.collate_fn(data) File "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py",
> line 79, in default_collate
> return [default_collate(samples) for samples in transposed] File "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py",
> line 79, in <listcomp>
> return [default_collate(samples) for samples in transposed] File "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py",
> line 55, in default_collate
> return torch.stack(batch, 0, out=out) TypeError: expected Tensor as element 3 in argument 0, but got tuple
我猜这两个数据集有时返回不同的类型。当数据是张量时,火炬把它们叠加起来,它们最好是相同的形状。如果它们有点像字符串,torch会把它们做成元组。这听起来像是你的一个数据集有时返回的不是张量的东西。我会在数据集的输出上添加一些断言,以检查它是否在做您想要的事情,或者使用
pdb
进行深入研究如果我答对了你的问题:你有如下的train和dev集合(以及它们相应的装载机)
您希望将它们连接起来,以便使用train+dev作为培训数据,对吗?如果是这样,您只需拨打:
列车开发装载机是包含两组数据的装载机
现在,请确保数据具有相同的形状和类型,即相同数量的要素,或相同的类别/编号等
除了@Leopd的答案之外,还可以使用PyTorch提供的
collate_fn
function。其思想是,在collate_fn
中,您将定义示例应如何堆叠以生成一个批。由于您使用的是torch 1.3.1,请确保您看到的是documentation的正确版本让我知道这是否有帮助,或者您是否有任何后续问题:)
相关问题 更多 >
编程相关推荐