After OS Patching all the Datanodes nodes are up but not alive and not able to connect to Namenode and all the blocks are showing missing

[#1]

ISSUE : Did OS Patch on all their hosts on the cluster and after that Datanodes are started but none of them are alive in the HDFS Dashboard and all the blocks are showing missing

Linux cl-master1 2.6.32-696.3.2.el6.x86_64 #1 SMP Wed Jun 7 11:51:39 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

RESOLUTION :

This is a Kernel Bug https://issues.apache.org/jira/browse/HDFS-12029

Please put the Namenode in Safe mode using the below command

Hdfs dfsadmin –safemode enter

Then add “-Xss2m” on in between the below lines

/usr/hdp/2.6.0.3-8/hadoop-hdfs/bin/hdfs.distro on all the Datanodes

exec "$JSVC" \
-Xss2m \
-Dproc_$COMMAND -outfile "$JSVC_OUTFILE" \
           -errfile "$JSVC_ERRFILE" \
           -pidfile "$HADOOP_SECURE_DN_PID" \
           -nodetach \
           -user "$HADOOP_SECURE_DN_USER" \
            -cp "$CLASSPATH" \
           $JAVA_HEAP_MAX $HADOOP_OPTS \
           org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter "$@"

Please restart the data nodes from Ambari and leave the safe mode on Namenode.

hdfs dfsadmin -leave safemode