我正在使用Spark Submit命令执行一个pyspark作业。之前它是工作文件,我已经执行了10多次相同的作业。它只是一个简单的从csv文件到配置单元表的数据加载命令,只包含500条记录。当我执行相同的命令时,现在它显示顶点失败问题
我正在使用下面的Spark提交命令
spark-submit --num-executors 3 --executor-cores 3 --executor-memory 20g
--jars /usr/hdp/3.1.0.0-78/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.1.0.0-78.jar
--py-files /usr/hdp/current/hive_warehouse_connector/pyspark_hwc-1.0.0.3.1.0.0-78.zip main.py
/user/hive/source
/user/hive/sourc>Source File Location
我收到下面的错误消息
Error while processing statement: FAILED: Execution Error,
return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1,
vertexId=vertex_1599711935259_0207_17_00, diagnostics=[Task failed, taskId=task_1599711935259_0207_17_00_000000,
diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) :
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0,
Vertex vertex_1500011935259_0207_17_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]Vertex killed,
vertexName=Reducer 2, vertexId=vertex_1500011935259_0207_17_00,
diagnostics=[Vertex received Kill while in RUNNING state.,
Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1,
Vertex vertex_1500011935259_0207_17_00 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1
有人能建议我如何解决这个错误吗
这个问题得到了解决,这是由于以下参数的计算不匹配
相关问题 更多 >
编程相关推荐