Namenode shutdown unexpectedly.

[#1]

what could be reason of namenode crash. Did not observe any GC or network issue. What can i check to find out possible causes.

2019-11-12 16:35:04,165 ERROR namenode.NameNode (LogAdapter.java:error(75)) - RECEIVED SIGNAL 15: SIGTERM
2019-11-12 16:35:04,167 INFO namenode.NameNode (LogAdapter.java:info(51)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hostname.com/ip
************************************************************/
2019-11-12 16:35:22,563 INFO namenode.NameNode (LogAdapter.java:info(51)) - STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = hosname/ip
STARTUP_MSG: args = []
STARTUP_MSG: version = 3.1.1.3.1.0.0-78

[#2]

@deekshant 

Can you share these 2 files the .out  logs the start process and the .log is the most interesting

/var/log/hadoop/hdfs/hadoop-hdfs-namenode-<host>.log
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-<host>.out

Please revert

[#3]

@deekshant ,

Please also check in the JournalNode log to see if there is any messages related at the time.

Cheers
Eric

[#4]

@deekshant 

To debug Namenode issue you need to check below -

1. Check active namenode[NN] logs [for time when it got reboot]

2. Check active NN zkfc logs [same time - if you see any issue]

3. Check for standby NN logs at same time if you see any error

4. Check for standby NN zkfc logs for any error at same timestamp

5. Check for Active NN .out file for any warnings/error

6. Check for system logs "/var/log/message" for any issue at particular moment of time.

You will find error in one of the above file. accordingly you can go for RCA.

Do revert if you need further help.