I am working on structured streaming. My job is running fine in Dev for past more than a week. But the same job is failing in PROD after running after few hours.
Two executors with Two core's each.
Source and Target: Kafka
Input rate : < 50 messages per second.
Few things I suspect for Yarn application log. let me know if you need more info. The full log file is more than 60 MB.
Caused by: org.apache.spark.SparkException: Job 4088 cancelled because SparkContext was shut down
Caused by: org.apache.spark.SparkException: Writing job aborted.
21:32:54.691 [dispatcher-event-loop-1] INFO o.a.s.e.CoarseGrainedExecutorBackend - Driver commanded a shutdown
21:32:54.568 [SIGTERM handler] ERROR o.a.s.deploy.yarn.ApplicationMaster - RECEIVED SIGNAL TERM
Thanks in advance.