我试图使用StringIndexer
、OneHotEncoder
和{
indexers = [
StringIndexer(inputCol=c, outputCol="{0}_indexed".format(c))
for c in columnList
]
encoders = [OneHotEncoder(dropLast=False, inputCol=indexer.getOutputCol(),
outputCol="{0}_encoded".format(indexer.getOutputCol()))
for indexer in indexers
]
assembler = VectorAssembler(inputCols=[encoder.getOutputCol() for encoder in encoders], outputCol="features")
pipeline = Pipeline(stages=indexers + encoders + [assembler])
model = pipeline.fit(df)
transformed = model.transform(df)
kmeans = KMeans().setK(2).setFeaturesCol("features").setPredictionCol("prediction")
kMeansPredictionModel = kmeans.fit(transformed)
predictionResult = kMeansPredictionModel.transform(transformed)
predictionResult.show(5)
我得到了Exception in thread "dag-scheduler-event-loop" java.lang.OutOfMemoryError: Java heap space
。如何在代码中分配更多堆空间或更好?分配更多空间是否明智?我可以限制程序的可用线程数和堆空间吗?在
我也遇到了同样的问题。用户允许的进程数不断增加。例如:
相关问题 更多 >
编程相关推荐