Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark Streaming job failed with below error after running fine for few hours.

Highlighted

Spark Streaming job failed with below error after running fine for few hours.

Explorer

Hi All

I am working on structured streaming. My job is running fine in Dev for past more than a week. But the same job is failing in PROD after running after few hours.

Configuration :

Two executors with Two core's each.

Source and Target: Kafka

Input rate : < 50 messages per second.

 

Few things I suspect for Yarn application log. let me know if you need more info. The full log file is more than 60 MB.

 

Any sugestions.

==============================================================

Caused by: org.apache.spark.SparkException: Job 4088 cancelled because SparkContext was shut down

Caused by: org.apache.spark.SparkException: Writing job aborted.

21:32:54.691 [dispatcher-event-loop-1] INFO o.a.s.e.CoarseGrainedExecutorBackend - Driver commanded a shutdown

21:32:54.568 [SIGTERM handler] ERROR o.a.s.deploy.yarn.ApplicationMaster - RECEIVED SIGNAL TERM

Thanks in advance.

Muthu

2 REPLIES 2

Re: Spark Streaming job failed with below error after running fine for few hours.

Explorer

Any update about this issue, I caught the same

Re: Spark Streaming job failed with below error after running fine for few hours.

Cloudera Employee

Hi Muthu,

 

The logs that has been shared is not sufficient. Could you please share the application master logs as you mentioned the log files seems to be huge?

 

 

Thanks

AK

Don't have an account?
Coming from Hortonworks? Activate your account here