RECEIVED SIGNAL 15: SIGTERM

[#1]

Hi all,

Spark streaming program will connect Hbase and ES to do some data verification realtime. But my spark application always crash after one or two days. And I got the following error. Can anybody tell me how to identify the reason ?

Thanks a lot.

16/12/18 19:08:24 ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL 15: SIGTERM
16/12/18 19:08:24 INFO common.GlobalHConnection: Closing the global HBase connection
16/12/18 19:08:24 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x458860dc3ec19c5
16/12/18 19:08:24 INFO storage.DiskBlockManager: Shutdown hook called
16/12/18 19:08:24 INFO zookeeper.ZooKeeper: Session: 0x458860dc3ec19c5 closed
16/12/18 19:08:24 INFO zookeeper.ClientCnxn: EventThread shut down
16/12/18 19:08:24 INFO common.GlobalHConnection: The global HBase connection closed

-----------------------------
Best Regards

[#2]

Is that log snippet from the AM or task logs?  Have you checked the other?  I have seen this message on the AM log side when something has terminated a task and the task log contains the detail as to why.

Is pre-emption enabled?  This message will be received, on the AM side, when tasks are killed by YARN when pre-emption kicks in.  Pre-emption works on the container level and there is always the risk that an AM container is killed in the process.  You could run the driver on the local client, --deploy-mode client, you launch from if you aren't already.

Turn up the logging to DEBUG.  Either in the Spark Gateway or by passing your own log4j.properties to the application.

Also check if spark.dynamicAllocation.enabled is on as I have seen sigterm 15 messages when it trims down the containers.  Cloudera also does not recommend having it turned on.