Hello I have to restart my HDFS but it keeps all time in safe mode, I make test to my datanodes and the communication is not the problem, It says capacity 0 live nodes 0 dead nodes 0 and is in safe mode, Is there a way to make the node alive?
I hope you can help me, thank you so much.
Created on 11-12-2019 08:39 PM - edited 11-12-2019 08:42 PM
@fgarciaCan you try to hit rest call and check if you get same info ?
curl -X GET admin: http://<active_namenode>:50070/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo
From the Namenode screenshot it seems that 0 datanodes/blocks are reported to NN.
Do you see all connections between DN and NN are good?
Can you check/pass full log strack?
Created 11-12-2019 08:56 PM
Can you try to leave the safemode.
And see if the communication is resumed back from datanodes to Name node.
You can use below command.
hadoop dfsadmin -safemode leave
Created 11-12-2019 09:26 PM
Just leave the safe mode by using
"hdfs dfsadmin - safemode leave" command.
Because Safe mode is read only mode. If it is on u can't write anything into hdfs. That's y cloudera manager or any distributions failed to create dir'r in hdfs.
Leave the safe mode and restart the hdfs once.
Created 11-12-2019 11:45 PM
I agree with @sagarshimpi , please provide the full log trace from the datanode and see if you are able to find any disk level exception in the logs are not. Also you can open NN UI-->"Datanode volume failures" tab and to confirm this.
Leaving safe mode manually will not solve the problem as NN did not receive any block reports from these data nodes, if you will leave the safe mode manually then you will get missing blocks alerts for all the blocks you have (approx 34 million blocks).
Did you change any configuration at hdfs level before restarting the whole HDFS service? If yes, then please share the property name.
Perform a disk i/o check on all the 4 data nodes using iostat command and check if disks are working fine without any heavy %util.