Support Questions
Find answers, ask questions, and share your expertise

HDFS gets 0 datanodes live but datanodes are started

Highlighted

HDFS gets 0 datanodes live but datanodes are started

New Contributor

Hello I have to restart my HDFS but it keeps all time in safe mode, I make test to my datanodes and the communication is not the problem, It says capacity 0 live nodes 0 dead nodes 0 and is in safe mode, Is there a way to make the node alive?

 

I hope you can help me, thank you so much.

4 REPLIES 4
Highlighted

Re: HDFS gets 0 datanodes live but datanodes are started

Expert Contributor

@fgarciaCan you try to hit rest call and check if you get same info ?

 

curl -X GET  admin: http://<active_namenode>:50070/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo

 

From the Namenode screenshot it seems that 0 datanodes/blocks are reported to NN.

Do you see all connections between DN and NN are good?

Can you check/pass full log strack?

 

 

Highlighted

Re: HDFS gets 0 datanodes live but datanodes are started

New Contributor

Can you try to leave the safemode.
And see if the communication is resumed back from datanodes to Name node.

You can use below command.

hadoop dfsadmin -safemode leave

Highlighted

Re: HDFS gets 0 datanodes live but datanodes are started

New Contributor

Just leave the safe mode by using

"hdfs dfsadmin - safemode leave" command. 

Because Safe mode is read only mode. If it is on u can't write anything into hdfs. That's y cloudera manager or any distributions failed to create dir'r in hdfs.

Leave the safe mode and restart the hdfs once. 

Highlighted

Re: HDFS gets 0 datanodes live but datanodes are started

Cloudera Employee

I agree with @sagarshimpi , please provide the full log trace from the datanode and see if you are able to find any disk level exception in the logs are not. Also you can open NN UI-->"Datanode volume failures" tab and to confirm this.
Leaving safe mode manually will not solve the problem as NN did not receive any block reports from these data nodes, if you will leave the safe mode manually then you will get missing blocks alerts for all the blocks you have (approx 34 million blocks).

Did you change any configuration at hdfs level before restarting the whole HDFS service? If yes, then please share the property name.
Perform a disk i/o check on all the 4 data nodes using iostat command and check if disks are working fine without any heavy %util.