文本分类数据集
textbook的Python项目详细描述
该框架以BERT为设计思想，目前支持七个常识推理数据集（alphanli，hellaswag，physicaliqa，socialiqa，codah，cosmosqa，和{
建筑
依赖关系
conda install av -c conda-forge^{pr2}$
下载原始数据集
./fetch.sh
它从AWS下载alphanli、hellaswag、physicaliqa、socialiqa、codah、cosmosqa、和{
使用
模板
模板的目标是将原始文本转换为中间数据，其中提供抽象信息供以后使用。在
理想情况下，模板应执行以下操作：
- 构造
text：一个列表列表。外部列表适用于多选择的情况，而内部列表则适用于每个输入对/三元组（例如上下文、问题和选择） - 构造
label：一个整数，表示一个表示真相的零索引标签，或者None - 构造
token_type_id和{}：段id和注意力的抽象表示。在下面的anli示例中， token_type_id和attention都有三个数字，分别代表文本每行的三个组件。在 - construct
image：以后要读取的任何形式的图像id/路径。在
安利的一个例子如下：
# rawcase={"story_id":"58090d3f-8a91-4c89-83ef-2b4994de9d241","obs1":"Ron started his new job as a landscaper today.","obs2":"Ron is immediately fired for insubordination.","hyp1":"Ron ignores his bosses's orders and called him an idiot.","hyp2":"Ron's boss called him an idiot.","label":"1"}# target intermediate datumtarget={'text':[['Ron started his new job as a landscaper today.',"Ron ignores his bosses's orders and called him an idiot.",'Ron is immediately fired for insubordination.'],['Ron started his new job as a landscaper today.',"Ron's boss called him an idiot.",'Ron is immediately fired for insubordination.']],'label':0,'image':None,'token_type_id':[0,1,0],'attention':[1,1,1]}LABEL2INT={"anli":{"1":0,"2":1,},}asserttemplate_anli(case,LABEL2INT['anli'])==target
渲染器
渲染器将中间基准转换为完全展开的基准。每个渲染器处理数据的不同部分。例如，renderer_text将文本呈现为input_id，并生成所有基于标记的attention和{renderer_video呈现image张量的image路径。呈现器以列表的形式传递给数据集构造函数，因此按顺序执行。在
BatchTool
我们提供了一个批处理工具，在这里可以很容易地使用传销或填充，您可以查看文档中的类以获取更多信息。在
加载包含pandas
fromtransformersimportBertTokenizerfromtextbookimportMultiModalDataset,template_anli,renderer_text,BatchTool,TokenBasedSamplerfromtorch.utils.dataimportDataset,DataLoaderfromtextbookimportLABEL2INTimportpandasaspdtokenizer=BertTokenizer.from_pretrained('bert-base-cased')d1=MultiModalDataset(df=pd.read_json("data_cache/alphanli/train.jsonl",lines=True),template=lambdax:template_anli(x,LABEL2INT['anli']),renderers=[lambdax:renderer_text(x,tokenizer)],)bt1=BatchTool(tokenizer,source="anli")i1=DataLoader(d1,batch_sampler=TokenBasedSampler(d1,batch_size=64),collate_fn=bt1.collate_fn)
使用多个数据集创建多任务数据集
fromtransformersimportBertTokenizerfromtextbookimport*importpandasaspdtokenizer=BertTokenizer.from_pretrained('bert-base-cased')# add additional tokens for each task as special `cls_token`tokenizer.add_special_tokens({"additional_special_tokens":["[ANLI]","[HELLASWAG]"]})d1=MultiModalDataset(df=pd.read_json("data_cache/alphanli/train.jsonl",lines=True),template=lambdax:template_anli(x,LABEL2INT['anli']),renderers=[lambdax:renderer_text(x,tokenizer,"[ANLI]")],)bt1=BatchTool(tokenizer,source="anli")i1=DataLoader(d1,batch_sampler=TokenBasedSampler(d1,batch_size=64),collate_fn=bt1.collate_fn)d2=MultiModalDataset(df=pd.read_json("data_cache/hellaswag/train.jsonl",lines=True),template=lambdax:template_hellaswag(x,LABEL2INT['hellaswag']),renderers=[lambdax:renderer_text(x,tokenizer,"[HELLASWAG]")],)bt2=BatchTool(tokenizer,source="hellaswag")i2=DataLoader(d2,batch_sampler=TokenBasedSampler(d1,batch_size=64),collate_fn=bt2.collate_fn)d=MultiTaskDataset([i1,i2],shuffle=False)#! batch size must be 1 for multitaskdataset, because we already batched in each sub dataset.forbatchinDataLoader(d,batch_size=1,collate_fn=BatchTool.uncollate_fn):pass# {# "source": "anli" or "hellaswag",# "labels": ...,# "input_ids": ...,# "attentions": ...,# "token_type_ids": ...,# "images": ...,# }
实现新模板或renderer
建议遵循以下约定，但是您可以做任何您想做的事情，因为您可以在任何地方调用lambda。在
deftemplate_xxx(raw_datum,*args,**kwargs):passdefrenderer_xxx(intermediate_datum,*args,**kwargs):pass
例如，对于Quora问题对数据集：
deftemplate_qqp(raw_datum,label2int={"0":0,"1":1},):result={"text":[[datum['question1'],datum['question2']]],"image":None,"label":Noneif'is_duplicate'notindatumordatum['is_duplicate']isNoneelselabel2int[str(datum['i_duplicate'])],"token_type_id":[0,1],"attention":[1,1],}returnresult
联系人
作者：牟承浩
- 项目
标签：
欢迎加入QQ群-->： 979659372
