- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
HDFS gets 0 datanodes live but datanodes are started
- Labels:
-
Apache Ambari
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello I have to restart my HDFS but it keeps all time in safe mode, I make test to my datanodes and the communication is not the problem, It says capacity 0 live nodes 0 dead nodes 0 and is in safe mode, Is there a way to make the node alive?
I hope you can help me, thank you so much.
Created on ‎11-12-2019 08:39 PM - edited ‎11-12-2019 08:42 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@fgarciaCan you try to hit rest call and check if you get same info ?
curl -X GET admin: http://<active_namenode>:50070/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo
From the Namenode screenshot it seems that 0 datanodes/blocks are reported to NN.
Do you see all connections between DN and NN are good?
Can you check/pass full log strack?
Created ‎11-12-2019 08:56 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can you try to leave the safemode.
And see if the communication is resumed back from datanodes to Name node.
You can use below command.
hadoop dfsadmin -safemode leave
Created ‎11-12-2019 09:26 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Just leave the safe mode by using
"hdfs dfsadmin - safemode leave" command.
Because Safe mode is read only mode. If it is on u can't write anything into hdfs. That's y cloudera manager or any distributions failed to create dir'r in hdfs.
Leave the safe mode and restart the hdfs once.
Created ‎11-12-2019 11:45 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I agree with @sagarshimpi , please provide the full log trace from the datanode and see if you are able to find any disk level exception in the logs are not. Also you can open NN UI-->"Datanode volume failures" tab and to confirm this.
Leaving safe mode manually will not solve the problem as NN did not receive any block reports from these data nodes, if you will leave the safe mode manually then you will get missing blocks alerts for all the blocks you have (approx 34 million blocks).
Did you change any configuration at hdfs level before restarting the whole HDFS service? If yes, then please share the property name.
Perform a disk i/o check on all the 4 data nodes using iostat command and check if disks are working fine without any heavy %util.
