无法在数据框上启动上下文，pysparkling

2024-09-30 01:33:46 发布

您现在位置：Python中文网/ 问答频道 /正文
6001 3

我试图在databricks集群上的python笔记簿中创建上下文。以下是我的环境规格：

  • Databricks运行时环境：5.3
  • 火花=2.4
  • Python=3.5
  • 颜色A>；=0.3.8
  • h2o-PY起泡剂-2.4

我正在编写以下代码：

from pysparkling import * from pyspark.sql import SparkSession import h2o spark = SparkSession.builder.appName("SparklingWaterApp").getOrCreate() h2oConf = H2OConf(spark).set("spark.ui.enabled", "false") hc = H2OContext.getOrCreate(spark, conf=h2oConf)`

我得到这个错误：

org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 1.0 failed 4 times, most recent failure: Lost task 2.3 in stage 1.0 (TID 40, 10.139.64.7, executor 2): java.io.InvalidClassException: org.apache.spark.h2o.backends.internal.InternalBackendUtils$; local class incompatible: stream classdesc serialVersionUID = -279081412540759760, local class serialVersionUID = -4513453206774459154 Py4JJavaError Traceback (most recent call last) in () 4 spark = SparkSession.builder.appName("SparklingWaterApp").getOrCreate() 5 h2oConf = H2OConf(spark).set("spark.ui.enabled", "false") ----> 6 hc = H2OContext.getOrCreate(spark, conf=h2oConf) /databricks/python/lib/python3.5/site-packages/pysparkling/context.py in getOrCreate(spark, conf, verbose, pre_create_hook, h2o_connect_hook, **kwargs) 159 160 # Create backing Java H2OContext --> 161 jhc = jvm.org.apache.spark.h2o.JavaH2OContext.getOrCreate(jspark_session, selected_conf._jconf) 162 h2o_context._jhc = jhc 163 h2o_context._conf = selected_conf /databricks/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in call(self, *args) 1255 answer = self.gateway_client.send_command(command) 1256 return_value = get_return_value( -> 1257 answer, self.gateway_client, self.target_id, self.name) 1258 1259 for temp_arg in temp_args:

请建议。你知道吗

谨致问候， 巴拉特


Tags： inorgimportselfapacheconfcontextstage
1条回答
网友
1楼 · 发布于 2024-09-30 01:33:46

请安装一个版本的sparkling\u water\u core\u 2\u 11\u 2\u 4\u x-xxxxx.jar文件与databricks运行时中安装的h2o版本相匹配。你知道吗

相关问题 更多 >

    编程相关推荐

    热门问题