live datanodes are limited to two, keep changing itself, but can't live more than two nodes

[#1]

Hi everyone,

I am facing very strange behaviour of live data nodes. I have hdp 2.6.4 running with four data nodes. At a time, only two datanodes can be live. I search in datanode logs no issue was showing there.

When I check the webui of namenodes for sometime then i figure out that the live nodes are changing itself after sometime. in my case, node5 to node-8 are datanodes. sometime node-5 and node6 are live then after sometime node-6 and node8 are live and keep changing after sometime.

How can I solve this issue.

--

Thanks

Amit Bondwal

[#2]

Now I added 8 data nodes, only five of them are live. Still facing the same issue, when I check the namenode web ui to check the datanodes, still datanodes are changing itself after sometime, but total number of live datanode is 5 out of 8. Any help or guide will be helpful to troubleshoot this issue.

see the namenode web ui, in pic one it is showing node-7 is live and pic-2 it is showing node-8 is live.

[#3]

I am continuously getting this type message in namenode logs, adding and removing nodes

2018-03-07 07:59:11,223 INFO net.NetworkTopology (NetworkTopology.java:add(427)) - Adding a new node: /default-rack/10.128.0.1:50010 2018-03-07 07:59:11,223 INFO blockmanagement.BlockReportLeaseManager (BlockReportLeaseManager.java:registerNode(205)) - Registered DN eac46cc3-a4e6-47e3-a15e-114a298da53e (10.128.0.1:50010). 2018-03-07 07:59:11,224 INFO blockmanagement.DatanodeDescriptor (DatanodeDescriptor.java:updateHeartbeatState(401)) - Number of failed storage changes from 0 to 0 2018-03-07 07:59:11,224 INFO blockmanagement.DatanodeDescriptor (DatanodeDescriptor.java:updateStorage(854)) - Adding new storage ID DS-1122bdfc-ff24-43e4-9669-b8fb587dc568 for DN 10.128.0.1:50010 2018-03-07 07:59:11,390 INFO hdfs.StateChange (DatanodeManager.java:registerDatanode(954)) - BLOCK* registerDatanode: from DatanodeRegistration(10.128.0.1:50010, datanodeUuid=56f106cc-cfb6-421f-b9fc-024a84a89c14, infoPort=50075, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-56;cid=CID-08d20112-9269-47cc-a86d-4e213d221aad;nsid=935392924;c=0) storage 56f106cc-cfb6-421f-b9fc-024a84a89c14 2018-03-07 07:59:11,390 INFO namenode.NameNode (DatanodeManager.java:registerDatanode(962)) - BLOCK* registerDatanode: 10.128.0.1:50010 2018-03-07 07:59:11,390 INFO net.NetworkTopology (NetworkTopology.java:remove(501)) - Removing a node: /default-rack/10.128.0.1:50010 2018-03-07 07:59:11,390 INFO blockmanagement.DatanodeDescriptor (DatanodeDescriptor.java:updateHeartbeatState(401)) - Number of failed storage changes from 0 to 0