NameNode safemode issue

[#1]

Not able to take Namenode off of SafeMode. Below is what happened:

a.) Ambari instructed to take HDFS checkpointThe
last HDFS checkpoint is older than 12 hours. Make sure that you have taken a
checkpoint before proceedingb.) 
b.) I then put the namenode in safemode and created a checkpoint
sudo su hdfs -l -c 'hdfs dfsadmin -safemode enter'
sudo su hdfs -l -c 'hdfs dfsadmin -saveNamespace'
3.) After that I am not able to restart nameNode. I tried to manually leave the safenode by trying the command: hdfs dfsadmin -safemode leave
4.) I also restarted all the datanode but nameNode is not restarting. it times out 
5.) I am using Ambari
Thanks

[#2]

Are you able to access NameNode web UI?

If yes then what message you see on webpage?

If not then please paste the contents of latest namenode service logs. Should be at /var/log/hadoop/hdfs

[#3]

@Rahul Pathak

I changed the "dfs.namenode.safemode.threshold-pct" from 1 to .9 but it gets changed to "1".

This is the information from the Namenode web UI:

Safe mode is ON. The reported blocks 1628 needs additional 2 blocks to reach the threshold 1.0000 of total blocks 1629. The number of live datanodes 4 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.

[#4]

@Prakash Punj

Are you changing dfs.namenode.safemode.threshold-pct from Ambari web UI?

If not then do it from Ambari web UI.

Does hdfs dfsadmin -safemode leave work?

Are you running above command from hdfs user?

Does hdfs dfsadmin -safemode leave give any error?

[#5]

last few lines of namenode log is

The reported blocks 1628 needs additional 2 blocks to reach the threshold 1.0000 of total blocks 1629.
The number of live datanodes 4 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
2016-03-15 13:39:28,096 INFO  ipc.Server (Server.java:run(2165)) - IPC Server handler 193 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.mkdirs from 10.0.2.23:47933 Call#0 Retry#229: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /system/yarn/node-labels. Name node is in safe mode.
The reported blocks 1628 needs additional 2 blocks to reach the threshold 1.0000 of total blocks 1629.
The number of live datanodes 4 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
2016-03-15 13:39:30,097 INFO  ipc.Server (Server.java:run(2165)) - IPC Server handler 193 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.mkdirs from 10.0.2.23:47933 Call#0 Retry#230: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /system/yarn/node-labels. Name node is in safe mode.
The reported blocks 1628 needs additional 2 blocks to reach the threshold 1.0000 of total blocks 1629.
The number of live datanodes 4 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.

[#6]

@Prakash Punj

Have you addressed this issue?

[#7 · Accepted solution]

@Prakash Punj

You need to take the NameNode out of the Safe Mode. This is a common error encountered when the NameNode is moved to Safe Mode to take a checkpoint. Documentation instructions should probably be updated to add a warning.

sudo su hdfs -l -c 'hdfs dfsadmin -safemode leave'

[#8]

@Prakash Punj

I just tested it. It is easy to reproduce. Stop YARN. Move the NameNode in Safe Mode and take a checkpoint. Attempt to Start YARN. It won't and the error is exactly the one described in the article. Take NameNode out of Safe Mode. Start YARN. All is good now!

Please vote/accept.