如何使用Pyspark解决sparksubmit上的spark processnext()错误

2024-06-28 14:44:57 发布

您现在位置:Python中文网/ 问答频道 /正文

我正在运行spark提交Pyspark作业,最近经常遇到这个错误。有人知道这是否可以通过设置解决吗

ERROR codegen.CodeGenerator: failed to compile: org.codehaus.janino.InternalCompilerException: Compiling "GeneratedClass": Code of method "processNext
()V" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1" grows beyond 64 KB org.codehaus.janino.InternalCompilerException: Compiling "GeneratedClass": Code of method "processNext()V" of class "org.apache.spark.sql.catalyst.expressions.Generated
Class$GeneratedIteratorForCodegenStage1" grows beyond 64 KB


Tags: oforgsqlapachecodemethodclassspark