- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
terminated with error org.apache.spark.SparkException: Job aborted due to stage failure: ResultStage
- Labels:
-
Apache Spark
Created on 04-18-2024 03:39 AM - edited 04-18-2024 03:40 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
While executing the spark application in yarn i was facing the following issue twice or thrice a day these days.
I suppose, i was casused when executing the following code:
df.writeStream
.foreachBatch { (batchDF: DataFrame, batchId: Long) =>
batchDF.persist()
------mongo insert 1---------
------mongo insert 2---------
batchDF
.select(DeviceIOIDS.TENANTGROUPUID.value, DeviceIOIDS.WORKERID.value, DeviceIOIDS.TASKUID.value)
.groupBy(DeviceIOIDS.TENANTGROUPUID.value, DeviceIOIDS.WORKERID.value, DeviceIOIDS.TASKUID.value)
.count()
.withColumnRenamed("count", "recordcount")
.withColumn(DeviceIOIDS.ISTRANSFERRED.value, lit(0))
.withColumn(DeviceIOIDS.INSERTDATETIME.value, current_timestamp())
.withColumn(DeviceIOIDS.INSERTDATE.value, current_date())
.write
.mode("Append")
.mongo(
WriteConfig(
"mongo.dbname".getConfigValue,
"mongo.devicelocationtransferredstatus".getConfigValue
)
)
batchDF.unpersist()
}
Thanks for the help
Created 04-21-2024 11:04 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @nagababu
There are couples of issues reported for similar kind of behaviour i.e SPARK-34790 , SPARK-18105 , SPARK-32658
You can try the following things:
Step1: Change the compression codec and run the application. For example, spark.io.compression.codec=snappy
Step2: If step1 is not resolved then try to set spark.file.transferTo=false and rerun the application.
Step3: You can set the following parameter and rerun the application
--conf spark.sql.adaptive.fetchShuffleBlocksInBatch=false
Step4: If any of the above steps are not resolved your issue then you can set the following parameters true and false and rerun the application.
spark.network.crypto.enabled=true
spark.authenticate=true
spark.io.encryption.enabled=true
Step5: If any of the above steps are not resolved your issue needs to tune the shuffle operation.
Created 04-18-2024 10:11 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@nagababu, Welcome to our community! To help you get the best possible answer, I have tagged in our Spark experts @RangaReddy who may be able to assist you further.
Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.
Regards,
Vidya Sargur,Community Manager
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:
Created 04-21-2024 11:04 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @nagababu
There are couples of issues reported for similar kind of behaviour i.e SPARK-34790 , SPARK-18105 , SPARK-32658
You can try the following things:
Step1: Change the compression codec and run the application. For example, spark.io.compression.codec=snappy
Step2: If step1 is not resolved then try to set spark.file.transferTo=false and rerun the application.
Step3: You can set the following parameter and rerun the application
--conf spark.sql.adaptive.fetchShuffleBlocksInBatch=false
Step4: If any of the above steps are not resolved your issue then you can set the following parameters true and false and rerun the application.
spark.network.crypto.enabled=true
spark.authenticate=true
spark.io.encryption.enabled=true
Step5: If any of the above steps are not resolved your issue needs to tune the shuffle operation.
Created on 06-17-2024 10:10 PM - edited 06-17-2024 10:12 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @RangaReddy , thank you very much for your response and suggestions. I tried the steps you recommended, and while they were helpful, I found that the issue was ultimately resolved by increasing the executor memory and by setting the spark.file.transferTo=false.
I appreciate your assistance.
Created 04-28-2024 11:31 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@nagababu, Did the response assist in resolving your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future.
Regards,
Vidya Sargur,Community Manager
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:
