Support Questions

Find answers, ask questions, and share your expertise
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Spark Streaming job failed with below error after running fine for few hours.


Hi All

I am working on structured streaming. My job is running fine in Dev for past more than a week. But the same job is failing in PROD after running after few hours.

Configuration :

Two executors with Two core's each.

Source and Target: Kafka

Input rate : < 50 messages per second.


Few things I suspect for Yarn application log. let me know if you need more info. The full log file is more than 60 MB.


Any sugestions.


Caused by: org.apache.spark.SparkException: Job 4088 cancelled because SparkContext was shut down

Caused by: org.apache.spark.SparkException: Writing job aborted.

21:32:54.691 [dispatcher-event-loop-1] INFO o.a.s.e.CoarseGrainedExecutorBackend - Driver commanded a shutdown

21:32:54.568 [SIGTERM handler] ERROR o.a.s.deploy.yarn.ApplicationMaster - RECEIVED SIGNAL TERM

Thanks in advance.




Any update about this issue, I caught the same

Cloudera Employee

Hi Muthu,


The logs that has been shared is not sufficient. Could you please share the application master logs as you mentioned the log files seems to be huge?





Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.