使用机器/深度学习算法可视化/训练/预测数据的python包
deepmg的Python项目详细描述
met2img(deepmg):使用深度学习的图像的亚基因组数据
met2img(deepmg)是使用深度学习和经典学习算法进行亚基因组分析的计算框架: (自2019年4月26日起(1.0.0及更高版本)转换为蟒蛇3)
- 支持将数据可视化成二维图像,使用多种不同算法训练1d或2d形状的数据,并使用预先训练的网络预测新数据。
- 提供多种存储空间:SPB、QTF、MMS,…
- 支持多种可视化数据的方法,包括填充、t分布随机邻域嵌入(t-sne)、线性判别分析(lda)、isomap、主成分分析(pca)、随机投影(rd-pro)、多维标度(mds)、谱嵌入。(SE)、非负矩阵分解(NMF)、局部线性嵌入(LLE)。
- 提供大量分类器(卷积神经网络、线性回归、随机森林、支持向量机、k近邻、梯度增强、k近邻……也可以从预先训练好的网络中加载,并且可以轻松地扩展)1D和2D数据。
- 包括交叉验证分析、内部验证和外部验证(可选)以及保留验证。
- 支持在使用PCA、RD U Pro可视化之前减少维度(对于非常维度的数据)
- 提供了测试具有大范围参数的模型的灵活性。
- 支持不同的数据类型,如丰度和不同级别的otu读取计数,如物种、属……
- 用不同的指标评估模型:准确性(acc)、曲线下面积(auc)、马修斯相关系数(mcc)、f1评分、混淆矩阵,…
- 25个可用数据集,包含5000个测试样本(下载自met2img,deepmg)
- 该软件包现在可以通过pip命令安装,支持macos、linux
- met2bin是生成1D表示的扩展版本。
参考文献:
如果有助于你的研究,请在你的出版物中引用met2img(deepmg)。非常感谢!
@article{deepmg_tsne,
author = {Thanh Hai Nguyen and
Yann Chevaleyre and
Edi Prifti and
Nataliya Sokolovska and
Jean{-}Daniel Zucker},
title = {Deep Learning for Metagenomic Data: using 2D Embeddings and Convolutional
Neural Networks},
journal = {CoRR, NIPS 2017 Workshop on Machine Learning for Health},
volume = {abs/1712.00244},
year = {2017},
url = {http://arxiv.org/abs/1712.00244},
archivePrefix = {arXiv},
biburl = {https://dblp.org/rec/bib/journals/corr/abs-1712-00244},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
入门
先决条件
- 如果没有,请安装:
python3 -m pip install numpy
python3 -m pip install matplotlib
python3 -m pip install ConfigParser
python3 -m pip install pandas
python3 -m pip install sklearn
python3 -m pip install tensorflow
python3 -m pip install keras
python3 -m pip install keras_sequential_ascii
- 为了使用这些软件包解释所培训的网络,请下载并安装:
[梯度凸轮,显著](https://github.com/jacobgil/keras梯度凸轮)
[石灰](https://github.com/marcotcr/lime/tree/master/doc/notebooks" rel="nofollow">https://github.com/marcotcr/lime/tree/master/doc/notebooks)
安装或下载软件包met2img
以便安装软件包
pip install deepmg
以便下载软件包
git clone https://git.integromics.fr/published/deepmg
运行实验
如何使用met2img
输入:
- 必需:包含数据(*u x.csv)和标签(*u y.csv)的csv文件
- 可选:如果使用外部验证集:数据(*z_x.csv)和标签(*z_y.csv))放入数据中,可使用参数进行更改--原始数据文件夹。
例如,[metaml]中肝硬化数据集的cirphy_x.csv和cirphy_y.csv(http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004977" rel="nofollow">https://journals.plos.org/ploscopbiol/article?id=10.1371/journal.pcbi.1004977)仅用于内部验证;和ibdtrainhs_ucr_x.csv ibdtrainhs_ucr_y.csv ibdtrainhs_ucrz_x.csv ibdtrainhs_ucrz_y.csv用于[Sokol's]中的数据集(https://www.ncbi.nlm.nih.gov/pubmed/26843508" rel="nofollow">https://www.ncbi.nlm.nih.gov/pubmed/26843508)包含外部验证集的数据集。
输出:
图像:met2img将生成图像并将其存储在[images/name_dataset_parameters_to_generate_image/](images/)(可使用参数进行更改--parent_folder_img)
结果:每个折叠的性能/培训/测试信息和摘要结果放入[结果/名称数据集_参数_生成_图像/](结果/)(可使用参数进行更改--父文件夹_结果),包含5个以上文件:
*file_sum.txt:用于运行的参数,每次折叠时的性能。最后一行显示了acc、auc、执行时间和其他实验指标的训练/测试性能。实验完成后,在标记实验完成的文件名后附加一个后缀"_ok"(可以用参数"suff fini"进行更改)。
*文件_each fold.txt(如果--save_folds=y):每一次折叠的准确结果、AUC、MCC、训练和测试损失。
*file_mean_acc.txt(如果--save_avg_run=y):如果实验包括独立重复的n运行,那么该文件包括每次运行的k倍上的平均性能,由精度和开始训练/测试时的时间执行来衡量。ng/完成后测试。
*file_mean_auc.txt(如果--save_avg_run=y):如果实验包括独立重复的n运行,则该文件包括每次运行的k-在开始的训练/测试、结束的训练/测试时由auc测量的倍数的平均性能。D.
如果要重复实验,请保存配置文件
如果使用--save_w=y(保存训练网络的权重)和/或--save_entire_w=y,--save_d=y,则将创建两个文件夹:
results/name_dataset_parameters_to_generate_image/models/:includes*weightmodel*.json包含模型的结构*weightmodel*.h5存储权重。
结果/name_dataset_parameters_to_generate_image/details/*weight挈txt:包含每个时期的准确性和训练和测试的损失。如果--save_rf=y,则每次运行时,我们都会从RFS中生成重要分数。
获取帮助以查看包中的参数:
Usage: dev_met2img.py [options]
Options:
-h, --help show this help message and exit
-a TYPE_RUN, --type_run=TYPE_RUN
Select a type of processing, visual (vis)
transformation of the data, learning (learn) a model,
testing it in another dataset, or predicting (predict)
new data, or creating config file (config)
--config_file=CONFIG_FILE
Specify the path of the config file if reading
parameters from files
--seed_value_begin=SEED_VALUE_BEGIN
set the beginning seed for different runs (default:1)
--search_already=SEARCH_ALREADY
if y: search existed experiments before running-->if
existed, stopping the experiment (default:y)
--cudaid=CUDAID id of the gpu card to use when multiple exist in the
server (if <=-3: use CPU, -2: use cpu if no available
gpu, -1: use all gpu, >=0: id of gpu), (default:-3)
--gpu_memory_fraction=GPU_MEMORY_FRACTION
gpu_memory_fraction for running by cuda
(0:auto_increase based on requirement)
--rd_pr_seed=RD_PR_SEED
seed for random projection (default:None)
--debug=DEBUG show DEBUG if >0 (default:0)
--check=CHECK check whether package can work properly or not (if
there is any error in installation)
--grid_coef_time=GRID_COEF_TIME
choose the best coef from #coef default for tuning
coeficiency (default:5)
--cv_coef_time=CV_COEF_TIME
k-cross validation for each coef for tuning
coeficiency (default:4)
--coef_ini=COEF_INI initilized coefficient for tuning coeficiency
(default:255)
--metric_selection=METRIC_SELECTION
roc_auc/accuracy/neg_log_loss/grid_search_mmc for
tuning coeficiency (default:roc_auc)
--parent_folder_img=PARENT_FOLDER_IMG
name of parent folder containing images
(default:images)
-r ORIGINAL_DATA_FOLDER, --original_data_folder=ORIGINAL_DATA_FOLDER
parent folder containing data (default:data)
-i DATA_NAME, --data_name=DATA_NAME
name of dataset (default:wt2phy)
--parent_folder_results=PARENT_FOLDER_RESULTS
parent folder containing results (default:results)
--save_avg_run=SAVE_AVG_RUN
save avg performance of each run
--save_labels=SAVE_LABELS
save labels of each fold
--save_d=SAVE_D saving details of learning (details of each epoch -->
may consume more storage
--save_w=SAVE_W save weight mode, default:n, this might consume much
storage
--suff_fini=SUFF_FINI
append suffix when finishing (default:ok), used to
mark finished experiment
--save_rf=SAVE_RF save important features and scores for Random Forests
--save_para=SAVE_PARA
save parameters to config files
--path_config_w=PATH_CONFIG_W
path of config to save, if empty, save to the same
folder of the results
--ext_data=EXT_DATA external data, default:
--save_entire_w=SAVE_ENTIRE_W
save weight of model on whole datasets (default:n)
--save_folds=SAVE_FOLDS
save results of each fold
--sound_fini=SOUND_FINI
play sound when finished (work on MacOS, default: n)
-k N_FOLDS, --n_folds=N_FOLDS
number of k folds (default:10), if k == 1, training on
whole *_x.csv, and testing on *z_x.csv
--test_ext=TEST_EXT if==y, using external validation sets (default:n)
--run_time=RUN_TIME give the #runs (default:1)
--whole_run_time=WHOLE_RUN_TIME
give the #runs (default:1)
--preprocess_img=PREPROCESS_IMG
support resnet50/vgg16/vgg19, none: no use
(default:none)
--pretrained_w_path=PRETRAINED_W_PATH
path to a pretrained model, used for testing,
predicting or continue to train a data
--test_size=TEST_SIZE
test size in holdout validation, if != 0 or != 1, then
do cross validation
-f NUMFILTERS, --numfilters=NUMFILTERS
#filters/neurons for each cnn/neural layer
-n NUMLAYERCNN_PER_MAXPOOL, --numlayercnn_per_maxpool=NUMLAYERCNN_PER_MAXPOOL
#cnnlayer before each max pooling (default:1)
--nummaxpool=NUMMAXPOOL
#maxpooling_layer (default:1)
--dropout_cnn=DROPOUT_CNN
dropout rate for CNN layer(s) (default:0)
-d DROPOUT_FC, --dropout_fc=DROPOUT_FC
dropout rate for FC layer(s) (default:0)
--padding=PADDING y if use pad, others: does not use (default:n)
--filtersize=FILTERSIZE
the filter size (default:3)
--poolsize=POOLSIZE the pooling size (default:2)
--model=MODEL model name for learning vgglike/model_lstm/resnet50/rf
_model/dtc_model/gbc_model/svm_model/knn_model/none/)
-c NUM_CLASSES, --num_classes=NUM_CLASSES
#output of the network (default:1)
-e EPOCH, --epoch=EPOCH
the epoch used for training (default:500)
--learning_rate=LEARNING_RATE
learning rate, if -1 use default value of the
optimizer (default:-1)
--batch_size=BATCH_SIZE
batch size (default:16)
--learning_rate_decay=LEARNING_RATE_DECAY
learning rate decay (default:0)
--momentum=MOMENTUM momentum (default:0)
-o OPTIMIZER, --optimizer=OPTIMIZER
support sgd/adam/Adamax/RMSprop/Adagrad/Adadelta/Nadam
(default:adam)
-l LOSS_FUNC, --loss_func=LOSS_FUNC
support binary_crossentropy/mae/squared_hinge/categori
cal_crossentropy (default:binary_crossentropy)
-q E_STOP, --e_stop=E_STOP
#epochs with no improvement after which training will
be stopped (default:5)
--e_stop_consec=E_STOP_CONSEC
option to choose consective (self defined: consec)
consec or norma, default:consec
--svm_c=SVM_C Penalty parameter C of the error term for SVM
--svm_kernel=SVM_KERNEL
the kernel type used in the algorithm (linear, poly,
rbf, sigmoid, precomputed) (default:linear)
--rf_n_estimators=RF_N_ESTIMATORS
The number of trees in the forest (default:500)
--rf_max_features=RF_MAX_FEATURES
The number of max features in the forest (default:-2:
auto (sqrt), -1: all)
--rf_max_depth=RF_MAX_DEPTH
The number of deep tree (default:-1: None)
--knn_n_neighbors=KNN_N_NEIGHBORS
The Number of neighbors to use (default:5) in
KNeighborsClassifier
-z COEFF, --coeff=COEFF
coeffiency (divided) for input (should use 255 for
images) (default:1)
--orderf_fill=ORDERF_FILL
shuffle order of feature (not for manifolds learning):
if none, use original order of data, if high: higher
composition will be in the top (default:none)
--new_dim=NEW_DIM new dimension after reduction (default:676)
--path_vis_learn=PATH_VIS_LEARN
read config for generating images (for qtf/mms combine
fills/manifolds)
--path_data_vis_learn=PATH_DATA_VIS_LEARN
path to read learned data for visualizing (for qtf/mms
combine fills/manifolds)
--del0=DEL0 if y, delete features have nothing
--lr_visual=LR_VISUAL
learning rate for generating visualizations
(default:100.0)
--label_visual=LABEL_VISUAL
use label when using t-SNE:'': does not use
--iter_visual=ITER_VISUAL
#iteration for run algorithm for visualization; for
tsne, it should be at least 250, but do not set so
high (default:300)
--ini_visual=INI_VISUAL
ini for visualization algorithm (default:pca)
--method_lle=METHOD_LLE
method for lle embedding:
standard/ltsa/hessian/modified (default:standard)
--eigen_solver=EIGEN_SOLVER
method for others (except for tsne) (default:auto)
--cmap_vmin=CMAP_VMIN
vmin for cmap (default:0)
--cmap_vmax=CMAP_VMAX
vmax for cmap (default:1)
--scale_mode=SCALE_MODE
scaler mode for input (default: Null)
--n_quantile=N_QUANTILE
n_quantile in quantiletransformer (default:1000)
--min_scale=MIN_SCALE
minimum value for scaling (only for minmaxscaler)
--max_scale=MAX_SCALE
maximum value for scaling (only for minmaxscaler)
--min_v=MIN_V limit min for Equal Width Binning (default:0)
--max_v=MAX_V limit max for Equal Width Binning (default:1), affect
for eqw ONLY
--num_bin=NUM_BIN the number of bins (default:10)
--auto_v=AUTO_V if = y, auto adjust min_v and max_v (default:0),
affect for eqw, and others in scaling
--mode_pre_img=MODE_PRE_IMG
support caffe/tf (default:caffe)
--channel=CHANNEL channel of images, 1: gray, 2: color (default:3)
-m DIM_IMG, --dim_img=DIM_IMG
width or height (square) of images, -1: get real size
of original images (default:-1)
-v VISUALIZE_MODEL, --visualize_model=VISUALIZE_MODEL
visualize the model if > 0 (default:0)
--algo_redu=ALGO_REDU
algorithm of dimension reduction (rd_pro/pca/fa), if
emtpy so do not use (default:)
--reduc_perle=REDUC_PERLE
perlexity for tsne (default:10)
--reduc_ini=REDUC_INI
ini for reduction (default:pca) of data (only use for
fillup) (default:none)
-t TYPE_EMB, --type_emb=TYPE_EMB
type of the embedding (default:raw):
raw/bin/fill/fills/zfill/zfills/tsne
--imp_fea=IMP_FEA using important features for overlapped issues
(default:none/rf/avg/max)
-g LABEL_EMB, --label_emb=LABEL_EMB
taxa level of labels provided in supervised embeddings
kingdom=1,phylum=2,class=3,order=4,family=5, genus=6
(default:0)
--emb_data=EMB_DATA data used for embbed with dimensionality reduction: :
transformed data; o: original data
-y TYPE_BIN, --type_bin=TYPE_BIN
type of binnings: spb/eqw/pr/eqf (default:)
-p PERLEXITY_NEIGHBOR, --perlexity_neighbor=PERLEXITY_NEIGHBOR
perlexity for tsne/#neighbors for others (default:5)
--n_components_emb=N_COMPONENTS_EMB
ouput after embedding (default:2)
-s SHAPE_DRAWN, --shape_drawn=SHAPE_DRAWN
shape of point to illustrate data:
(pixel)/ro/o(circle) (default:,)
--fig_size=FIG_SIZE fig_size to contain all features, if 0:smallest which
fit data, (default:0)
--point_size=POINT_SIZE
point size for img (default:1)
--colormap=COLORMAP colormaps for generating images (rainbow
/nipy_spectral/jet/Paired/Reds/YlGnBu)
(default:custom)
--margin=MARGIN margin to images (default:0)
--alpha_v=ALPHA_V 1 (opaque) (default:1)
--recreate_img=RECREATE_IMG
if >0 rerun to create images even though they are
existing (default:0)
下面是一些示例:
可用数据集
默认情况下,框架在10个分层交叉验证下运行10次
**注意:
- 选择RunonGPU,将CUDAID设置为(0,1,2,3)(机器上GPU的ID,ID=-1表示使用CPU)。注意:您的计算节点必须支持GPU并安装TensorFlow GPU。
- 选择参数为'-i'的数据集,例如'-i cirphy'(系统发育性肝硬化数据集)
- 选择带有参数'--model'的模型,例如'--model model\u cnn'。默认:具有一个完全连接层(FC)模型的模型
- 其他参数,请参见函数para_cmd() < > >
- 1d:运行具有1d数据的模型的脚本
- 流形iso:使用基于isomap的可视化训练物种丰富度。
- 多方面mds:使用基于mds的可视化训练物种丰富度。
- 流形nmf:使用基于nmf的可视化训练物种丰富度。
- 流形pca:使用基于pca的可视化训练物种丰富度。
- 流形lda1,2,3,4,5,6:使用基于lda(监督)的可视化训练物种丰富度,标签使用不同水平的otus(1:王国,2:门,3:类,4:顺序,5:科,6:属)。
- Phy0_24_CMAP_r:调查大量的彩色地图(Viridis、Rainbow、Jet,…)
- phyfill0_vgg:研究vgg架构的各种参数。
- fill0cnn:使用不同cnn超参数填充运行实验。
- phyFill0 rnd:使用填充随机特征排序的实验
- 基因填充:使用填充和机器学习算法(SVM、RF)训练基因家族丰度(名称:cirgene、colgene、ibdgene、obegene、t2dgene、wt2dgene)。
- PhyFill0_CRC:对论文中的数据集(Yu,Feng,Zeller,Vogtmann,CRC)进行的实验结直肠癌基因组的多队列分析确定了人群中的改变细菌。ONS和通用细菌
- phyfill0_phcnn:在论文中的数据集(文件:ibdtrainhs_cdf、ibdtrainhs_cdr、ibdtrainhs_icdf、ibdtrainhs_icdr、ibdtrainhs_ucf、ibdtrainhs_ucr)上的实验亚基因组学中的al神经网络
- balance_phyfill0(对于彩色图像)和balance_phygrayfill0(对于灰色图像):论文中的数据集(hiv,crohn)实验:平衡:微生物组分分析的新视角
- 比较流形.ipynb:从manifol生成的可视化d学习,如t-sne、lda、isomap
- plot_distribution_taxa_levels_colormaps.ipynb:使用fill-up演示fill-up的工作原理和可视化重要功能
- visual_fillup_colormaps.ipynb:演示各种颜色映射
- vis_cnn_lime_grad.ipynb解释:按显著性、lime和grad cam显示解释
- 读取结果.py:收集所有实验:
- 每行显示acc,auc,mcc,f1得分,执行时间…(results_sum.txt)
- 每一行显示一个关于acc,auc,mcc,混淆矩阵的实验结果。(results_folds.txt)
- 每行显示一个实验的外部验证结果,包括acc,auc,mcc,混淆矩阵…(结果文本)
- unf_delete.py:删除/计数未完成实验的未完成日志文件。
- Thanh Hai Nguyen(电子邮件:hainguyen579[at]gmail.com或nthai[at]cit.ctu.edu.vn)
- edi-prifti(电子邮件:e.prifti[at]ican institute.org)
- Nataliya Sokolovska(电子邮件:Nataliya.Sokolovska[at]upmc.fr)
- 让·丹尼尔·祖克(jean-daniel zucker)(电子邮件:jean-daniel.zucker[at]ird.fr)
运行实验的代码(用于原始数据)
参数:-n:卷积层数,-f:滤波器数, -T:嵌入类型(支持原始-1d和图像2d,如fillup、t-sne、isomap、lda等)
db='wt2dphy';
python3 -m deepmg -i $db -t raw
python3 -m deepmg -i $db -t raw --model model_cnn1d -n 1 -f 64
运行实验的代码(用于填充灰色图像)
使用SPB
db='wt2dphy';
python3 -m deepmg -i $db -t fill -y spb -z 255 --colormap gray --channel 1
python3 -m deepmg -i $db -t fill -y spb -z 255 --colormap gray --channel 1 --model model_cnn -n 1 -f 64
使用pr
db='wt2dphy';
python3 -m deepmg -i $db -t fill -y pr -z 255 --colormap gray --channel 1
使用qtf
db='wt2dphy';
python3 -m deepmg -i $db -t fills -y eqw --scale_mode qtf --auto_v y -z 255 --colormap gray --channel 1
运行实验的代码(用于填充彩色图像)
@article{deepmg_tsne,
author = {Thanh Hai Nguyen and
Yann Chevaleyre and
Edi Prifti and
Nataliya Sokolovska and
Jean{-}Daniel Zucker},
title = {Deep Learning for Metagenomic Data: using 2D Embeddings and Convolutional
Neural Networks},
journal = {CoRR, NIPS 2017 Workshop on Machine Learning for Health},
volume = {abs/1712.00244},
year = {2017},
url = {http://arxiv.org/abs/1712.00244},
archivePrefix = {arXiv},
biburl = {https://dblp.org/rec/bib/journals/corr/abs-1712-00244},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
0
基于流形学习的可视化实验代码,例如,t-sne(将'-t'的参数更改为'tsne')
我们还可以测试其他嵌入,如isomap,lle,… 我们使用24x24(--fig_大小24)和透明率(alpha_v=0.5)的图像
@article{deepmg_tsne,
author = {Thanh Hai Nguyen and
Yann Chevaleyre and
Edi Prifti and
Nataliya Sokolovska and
Jean{-}Daniel Zucker},
title = {Deep Learning for Metagenomic Data: using 2D Embeddings and Convolutional
Neural Networks},
journal = {CoRR, NIPS 2017 Workshop on Machine Learning for Health},
volume = {abs/1712.00244},
year = {2017},
url = {http://arxiv.org/abs/1712.00244},
archivePrefix = {arXiv},
biburl = {https://dblp.org/rec/bib/journals/corr/abs-1712-00244},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
1
脚本(sh)的详细信息,请参见utils/scripts,其中:
这些脚本主要用于a组(如果没有指定)的数据集,包括预测肝硬化、结直肠、ibd、肥胖、t2d(和wt2d数据集)。每个文件的头部分由内存、内核数、walltime、电子邮件等信息组成,这些信息用于作业调度程序。这些参数应根据可用资源进行修改。每个文件为一个数据集运行多个模型。
6个数据集的脚本(文件:cirphy_*(cirphy_x.csv用于数据,cirphy_y.csv用于标签,为了在该数据集上运行培训,请将参数-i设置为"-i cirphy")、colphy_*、ibdphy_*、obephy_*、t2dphy_*、wt2dphy_*)设置为a组[metaml](https://journals.plos.org/ploscopbiol/article?id=10.1371/journal.pcbi.1004977):
为其他组编写训练数据集脚本:
实用程序和可视化
用ascii可视化模型
只需添加"-v 1"即可可视化网络。要使用此功能,请安装"keras_sequential_ascii"
@article{deepmg_tsne,
author = {Thanh Hai Nguyen and
Yann Chevaleyre and
Edi Prifti and
Nataliya Sokolovska and
Jean{-}Daniel Zucker},
title = {Deep Learning for Metagenomic Data: using 2D Embeddings and Convolutional
Neural Networks},
journal = {CoRR, NIPS 2017 Workshop on Machine Learning for Health},
volume = {abs/1712.00244},
year = {2017},
url = {http://arxiv.org/abs/1712.00244},
archivePrefix = {arXiv},
biburl = {https://dblp.org/rec/bib/journals/corr/abs-1712-00244},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
2
jupyter:表示的可视化
请转到/utils/jupyter/以可视化基于图像的表示:
@article{deepmg_tsne,
author = {Thanh Hai Nguyen and
Yann Chevaleyre and
Edi Prifti and
Nataliya Sokolovska and
Jean{-}Daniel Zucker},
title = {Deep Learning for Metagenomic Data: using 2D Embeddings and Convolutional
Neural Networks},
journal = {CoRR, NIPS 2017 Workshop on Machine Learning for Health},
volume = {abs/1712.00244},
year = {2017},
url = {http://arxiv.org/abs/1712.00244},
archivePrefix = {arXiv},
biburl = {https://dblp.org/rec/bib/journals/corr/abs-1712-00244},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
3
总结结果
此项目中提供了一些工具,支持收集数据、筛选结果和删除未完成的实验