<p>我有一个spark作业,它用下面的代码运行文件。但是,此步骤将在输出文件夹中创建多个文件。你知道吗</p>
<pre><code>sampledataframe.write.mode('append').partitionBy('DATE_FIELD').save(FILEPATH)
</code></pre>
<p>因此,我开始使用下面的代码行重新划分这些文件,并有一个单一的文件。你知道吗</p>
<pre><code>sampledataframe.repartition('DATE_FIELD').write.mode('append').partitionBy('DATE_FIELD').save(FILEPATH)
</code></pre>
<p>这段代码在几个月内运行良好,但最近开始失败,出现以下错误。你知道吗</p>
<pre><code>[2019-09-26 16:15:48,030] {bash_operator.py:74} INFO - 19/09/26 16:15:48 WARN TaskSetManager: Lost task 48.0 in stage 1.0 (TID 812, aaa.bbb.io): org.apache.spark.SparkException: Task failed while writing rows
[2019-09-26 16:15:48,031] {bash_operator.py:74} INFO - at org.apache.spark.sql.execution.datasources.DynamicPartitionWriterContainer.writeRows(WriterContainer.scala:417)
[2019-09-26 16:15:48,031] {bash_operator.py:74} INFO - at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:148)
[2019-09-26 16:15:48,031] {bash_operator.py:74} INFO - at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:148)
[2019-09-26 16:15:48,031] {bash_operator.py:74} INFO - at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
[2019-09-26 16:15:48,031] {bash_operator.py:74} INFO - at org.apache.spark.scheduler.Task.run(Task.scala:89)
[2019-09-26 16:15:48,032] {bash_operator.py:74} INFO - at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:247)
[2019-09-26 16:15:48,032] {bash_operator.py:74} INFO - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2019-09-26 16:15:48,032] {bash_operator.py:74} INFO - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2019-09-26 16:15:48,032] {bash_operator.py:74} INFO - at java.lang.Thread.run(Thread.java:748)
[2019-09-26 16:15:48,032] {bash_operator.py:74} INFO - Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
</code></pre>
<p>以前有人遇到过这个错误吗?你能告诉我怎么解决这个问题吗?你知道吗</p>