Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

live datanodes are limited to two, keep changing itself, but can't live more than two nodes

New Contributor

Hi everyone,

I am facing very strange behaviour of live data nodes. I have hdp 2.6.4 running with four data nodes. At a time, only two datanodes can be live. I search in datanode logs no issue was showing there.

When I check the webui of namenodes for sometime then i figure out that the live nodes are changing itself after sometime. in my case, node5 to node-8 are datanodes. sometime node-5 and node6 are live then after sometime node-6 and node8 are live and keep changing after sometime.

How can I solve this issue.

--

Thanks

Amit Bondwal

2 REPLIES 2

New Contributor

Now I added 8 data nodes, only five of them are live. Still facing the same issue, when I check the namenode web ui to check the datanodes, still datanodes are changing itself after sometime, but total number of live datanode is 5 out of 8. Any help or guide will be helpful to troubleshoot this issue.

see the namenode web ui, in pic one it is showing node-7 is live and pic-2 it is showing node-8 is live.

64456-pic-1.png

64457-pic-2.png

New Contributor

I am continuously getting this type message in namenode logs, adding and removing nodes

2018-03-07 07:59:11,223 INFO net.NetworkTopology (NetworkTopology.java:add(427)) - Adding a new node: /default-rack/10.128.0.1:50010 2018-03-07 07:59:11,223 INFO blockmanagement.BlockReportLeaseManager (BlockReportLeaseManager.java:registerNode(205)) - Registered DN eac46cc3-a4e6-47e3-a15e-114a298da53e (10.128.0.1:50010). 2018-03-07 07:59:11,224 INFO blockmanagement.DatanodeDescriptor (DatanodeDescriptor.java:updateHeartbeatState(401)) - Number of failed storage changes from 0 to 0 2018-03-07 07:59:11,224 INFO blockmanagement.DatanodeDescriptor (DatanodeDescriptor.java:updateStorage(854)) - Adding new storage ID DS-1122bdfc-ff24-43e4-9669-b8fb587dc568 for DN 10.128.0.1:50010 2018-03-07 07:59:11,390 INFO hdfs.StateChange (DatanodeManager.java:registerDatanode(954)) - BLOCK* registerDatanode: from DatanodeRegistration(10.128.0.1:50010, datanodeUuid=56f106cc-cfb6-421f-b9fc-024a84a89c14, infoPort=50075, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-56;cid=CID-08d20112-9269-47cc-a86d-4e213d221aad;nsid=935392924;c=0) storage 56f106cc-cfb6-421f-b9fc-024a84a89c14 2018-03-07 07:59:11,390 INFO namenode.NameNode (DatanodeManager.java:registerDatanode(962)) - BLOCK* registerDatanode: 10.128.0.1:50010 2018-03-07 07:59:11,390 INFO net.NetworkTopology (NetworkTopology.java:remove(501)) - Removing a node: /default-rack/10.128.0.1:50010 2018-03-07 07:59:11,390 INFO blockmanagement.DatanodeDescriptor (DatanodeDescriptor.java:updateHeartbeatState(401)) - Number of failed storage changes from 0 to 0