I am working on structured streaming. My job is running fine in Dev for past more than a week. But the same job is failing in PROD after running after few hours.
Two executors with Two core's each.
Source and Target: Kafka
Input rate : < 50 messages per second.
Few things I suspect for Yarn application log. let me know if you need more info. The full log file is more than 60 MB.
Caused by: org.apache.spark.SparkException: Job 4088 cancelled because SparkContext was shut down
Caused by: org.apache.spark.SparkException: Writing job aborted.
21:32:54.691 [dispatcher-event-loop-1] INFO o.a.s.e.CoarseGrainedExecutorBackend - Driver commanded a shutdown
21:32:54.568 [SIGTERM handler] ERROR o.a.s.deploy.yarn.ApplicationMaster - RECEIVED SIGNAL TERM
Thanks in advance.
The logs that has been shared is not sufficient. Could you please share the application master logs as you mentioned the log files seems to be huge?