Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark Streaming job failed with below error after running fine for few hours.

Highlighted

Spark Streaming job failed with below error after running fine for few hours.

New Contributor

Hi All

I am working on structured streaming. My job is running fine in Dev for past more than a week. But the same job is failing in PROD after running after few hours.

Configuration :

Two executors with Two core's each.

Source and Target: Kafka

Input rate : < 50 messages per second.


Few things I suspect for Yarn application log. let me know if you need more info. The full log file is more than 60 MB.


Any sugestions.

==============================================================

Caused by: org.apache.spark.SparkException: Job 4088 cancelled because SparkContext was shut down

Caused by: org.apache.spark.SparkException: Writing job aborted.

21:32:54.691 [dispatcher-event-loop-1] INFO o.a.s.e.CoarseGrainedExecutorBackend - Driver commanded a shutdown

21:32:54.568 [SIGTERM handler] ERROR o.a.s.deploy.yarn.ApplicationMaster - RECEIVED SIGNAL TERM

Thanks in advance.

Muthu