bert在tpu上tensorflow包keras中的实现
keras-bert-tpu的Python项目详细描述
这是一个CyberZHG/keras_bert的叉，它支持tpu上的keras bert。
实现BERT。官方预先训练的模型可以加载到特征提取和预测。
COLAB演示
安装
pip install keras-bert-tpu
用法
加载官方预先培训的模型
在feature extraction demo中，您应该能够获得与官方模型相同的提取结果。在prediction demo中，可以预测句子中缺少的单词。
培训和使用
fromkeras_bertimportget_base_dict,get_model,gen_batch_inputs# A toy input examplesentence_pairs=[[['all','work','and','no','play'],['makes','jack','a','dull','boy']],[['from','the','day','forth'],['my','arm','changed']],[['and','a','voice','echoed'],['power','give','me','more','power']],]# Build token dictionarytoken_dict=get_base_dict()# A dict that contains some special tokensforpairsinsentence_pairs:fortokeninpairs[0]+pairs[1]:iftokennotintoken_dict:token_dict[token]=len(token_dict)token_list=list(token_dict.keys())# Used for selecting a random word# Build & train the modelmodel=get_model(token_num=len(token_dict),head_num=5,transformer_num=12,embed_dim=25,feed_forward_dim=100,seq_len=20,pos_num=20,dropout_rate=0.05,)model.summary()def_generator():whileTrue:yieldgen_batch_inputs(sentence_pairs,token_dict,token_list,seq_len=20,mask_rate=0.3,swap_sentence_rate=1.0,)model.fit_generator(generator=_generator(),steps_per_epoch=1000,epochs=100,validation_data=_generator(),validation_steps=100,callbacks=[keras.callbacks.EarlyStopping(monitor='val_loss',patience=5)],)# Use the trained modelinputs,output_layer=get_model(# `output_layer` is the last feature extraction layer (the last transformer)token_num=len(token_dict),head_num=5,transformer_num=12,embed_dim=25,feed_forward_dim=100,seq_len=20,pos_num=20,dropout_rate=0.05,training=False,# The input layers and output layer will be returned if `training` is `False`)
自定义特征提取
def_custom_layers(x,trainable=True):returnkeras.layers.LSTM(units=768,trainable=trainable,name='LSTM',)(x)model=get_model(token_num=200,embed_dim=768,custom_layers=_custom_layers,)
欢迎加入QQ群-->： 979659372
