"Spark随机森林交叉验证"

Question

我试图在Spark的一个随机林中运行交叉验证。在

from pyspark.ml import Pipeline
from pyspark.ml.classification import RandomForestClassifier
from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
from pyspark.ml.evaluation import MulticlassClassificationEvaluator

data = nds.sc.parallelize([
 LabeledPoint(0.0, [0,402,6,0]),
 LabeledPoint(0.0, [3,500,3,0]),
 LabeledPoint(1.0, [1,590,1,1]),
 LabeledPoint(1.0, [3,328,5,0]),
 LabeledPoint(1.0, [4,351,4,0]),
 LabeledPoint(0.0, [2,372,2,0]),
 LabeledPoint(0.0, [4,302,5,0]),
 LabeledPoint(1.0, [1,387,2,0]),
 LabeledPoint(1.0, [1,419,3,0]),
 LabeledPoint(0.0, [1,370,5,0]),
 LabeledPoint(0.0, [1,410,4,0]),
 LabeledPoint(0.0, [2,509,7,1]),
 LabeledPoint(0.0, [1,307,5,0]),
 LabeledPoint(0.0, [0,424,4,1]),
 LabeledPoint(0.0, [1,509,2,1]),
 LabeledPoint(1.0, [3,361,4,0]),
 ])


train=data.toDF(['label','features'])

numfolds =2

rf = RandomForestClassifier(labelCol="label", featuresCol="features")
evaluator = MulticlassClassificationEvaluator()  


paramGrid = ParamGridBuilder().addGrid(rf.maxDepth,      
[4,8,10]).addGrid(rf.impurity, ['entropy','gini']).addGrid(rf.featureSubsetStrategy, [6,8,10]).build()

pipeline = Pipeline(stages=[rf])

crossval = CrossValidator(
    estimator=pipeline,
    estimatorParamMaps=paramGrid,
    evaluator=evaluator,
    numFolds= numfolds)

model = crossval.fit(train)

我得到以下错误

^{pr2}$

似乎paramGrid没有将我的输入作为列表读取。是否有其他格式或解决方法。任何帮助都将不胜感激。在

Answer 1

向rf.featureSubsetStrategy传递了不正确的值。它应该是一个描述策略的字符串，它支持以下值：auto、all、onethird、sqrt、log2。请参见：RandomForestClassifier.featureSubsetStrategy.doc。在

也不要使用data.toDF(['label','features'])。它不能保持正确的秩序。使用：

data.toDF()

或者如果要修改名称：

^{pr2}$

最后，标签列必须被索引，或者您必须提供所需的元数据。见How can I declare a Column as a categorical feature in a DataFrame for use in ml