Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Datnode is not alive tough datanode is up

Datnode is not alive tough datanode is up

Explorer

We have 3 datanodes in and only 2 datanodes are alive tough 3 datanodes are up ,amoung 3 datanodes 2 datanodes are in default rack and 1 datanode is in specified rack ,datanode in specified rack is not going live

 

tied refreshing and restarting of entire hdfs ,nothing helped 

could see below error in logs 

 

2020-09-16 13:13:24,006 ERROR Error: can't add leaf node /AZ/EUS2/ambari/0/0.0.0.1:50010 at depth 6 to topology:Number of rcks: 1 Expected number of leaves:2 /default-rack/0.0.0.2:50010 /default-rack/0.0.0.3:50010
2020-09-16 13:13:24,007 WARN IPC Server handler 182 on 8020, call Call#419 Retry#0 org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol.registerDatanode from 0.0.0.4:54685 org.apache.hadoop.net.NetworkTopology$InvalidTopologyException: Failed to add /AZ/EUS2/ambari//0/0.0.0.1:50010: You cannot have a rack and a non-rack node at the same level of the network topology. at org.apache.hadoop.net.NetworkTopology.add(NetworkTopology.java:134) at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:1135) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.registerDatanode(BlockManager.java:2354) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:3830) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:1490) at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:101) at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:31658) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:268

Don't have an account?
Coming from Hortonworks? Activate your account here