Spark Streaming job failed with below error after running fine for few hours.

[#1]

Hi All

I am working on structured streaming. My job is running fine in Dev for past more than a week. But the same job is failing in PROD after running after few hours.

Configuration :

Two executors with Two core's each.

Source and Target: Kafka

Input rate : < 50 messages per second.

Few things I suspect for Yarn application log. let me know if you need more info. The full log file is more than 60 MB.

Any sugestions.

==============================================================

Caused by: org.apache.spark.SparkException: Job 4088 cancelled because SparkContext was shut down

Caused by: org.apache.spark.SparkException: Writing job aborted.

21:32:54.691 [dispatcher-event-loop-1] INFO o.a.s.e.CoarseGrainedExecutorBackend - Driver commanded a shutdown

21:32:54.568 [SIGTERM handler] ERROR o.a.s.deploy.yarn.ApplicationMaster - RECEIVED SIGNAL TERM

Thanks in advance.

Muthu

[#2]

Any update about this issue, I caught the same

[#3]

Hi Muthu,

The logs that has been shared is not sufficient. Could you please share the application master logs as you mentioned the log files seems to be huge?

Thanks

AK