Support Questions

Find answers, ask questions, and share your expertise

live datanodes are limited to two, keep changing itself, but can't live more than two nodes

New Contributor

Hi everyone,

I am facing very strange behaviour of live data nodes. I have hdp 2.6.4 running with four data nodes. At a time, only two datanodes can be live. I search in datanode logs no issue was showing there.

When I check the webui of namenodes for sometime then i figure out that the live nodes are changing itself after sometime. in my case, node5 to node-8 are datanodes. sometime node-5 and node6 are live then after sometime node-6 and node8 are live and keep changing after sometime.

How can I solve this issue.



Amit Bondwal


New Contributor

Now I added 8 data nodes, only five of them are live. Still facing the same issue, when I check the namenode web ui to check the datanodes, still datanodes are changing itself after sometime, but total number of live datanode is 5 out of 8. Any help or guide will be helpful to troubleshoot this issue.

see the namenode web ui, in pic one it is showing node-7 is live and pic-2 it is showing node-8 is live.



New Contributor

I am continuously getting this type message in namenode logs, adding and removing nodes

2018-03-07 07:59:11,223 INFO net.NetworkTopology ( - Adding a new node: /default-rack/ 2018-03-07 07:59:11,223 INFO blockmanagement.BlockReportLeaseManager ( - Registered DN eac46cc3-a4e6-47e3-a15e-114a298da53e ( 2018-03-07 07:59:11,224 INFO blockmanagement.DatanodeDescriptor ( - Number of failed storage changes from 0 to 0 2018-03-07 07:59:11,224 INFO blockmanagement.DatanodeDescriptor ( - Adding new storage ID DS-1122bdfc-ff24-43e4-9669-b8fb587dc568 for DN 2018-03-07 07:59:11,390 INFO hdfs.StateChange ( - BLOCK* registerDatanode: from DatanodeRegistration(, datanodeUuid=56f106cc-cfb6-421f-b9fc-024a84a89c14, infoPort=50075, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-56;cid=CID-08d20112-9269-47cc-a86d-4e213d221aad;nsid=935392924;c=0) storage 56f106cc-cfb6-421f-b9fc-024a84a89c14 2018-03-07 07:59:11,390 INFO namenode.NameNode ( - BLOCK* registerDatanode: 2018-03-07 07:59:11,390 INFO net.NetworkTopology ( - Removing a node: /default-rack/ 2018-03-07 07:59:11,390 INFO blockmanagement.DatanodeDescriptor ( - Number of failed storage changes from 0 to 0