Support Questions
Find answers, ask questions, and share your expertise

Datnode is not alive tough datanode is up

Datnode is not alive tough datanode is up

Explorer

We have 3 datanodes in and only 2 datanodes are alive tough 3 datanodes are up ,amoung 3 datanodes 2 datanodes are in default rack and 1 datanode is in specified rack ,datanode in specified rack is not going live

 

tied refreshing and restarting of entire hdfs ,nothing helped 

could see below error in logs 

 

2020-09-16 13:13:24,006 ERROR Error: can't add leaf node /AZ/EUS2/ambari/0/0.0.0.1:50010 at depth 6 to topology:Number of rcks: 1 Expected number of leaves:2 /default-rack/0.0.0.2:50010 /default-rack/0.0.0.3:50010
2020-09-16 13:13:24,007 WARN IPC Server handler 182 on 8020, call Call#419 Retry#0 org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol.registerDatanode from 0.0.0.4:54685 org.apache.hadoop.net.NetworkTopology$InvalidTopologyException: Failed to add /AZ/EUS2/ambari//0/0.0.0.1:50010: You cannot have a rack and a non-rack node at the same level of the network topology. at org.apache.hadoop.net.NetworkTopology.add(NetworkTopology.java:134) at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:1135) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.registerDatanode(BlockManager.java:2354) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:3830) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:1490) at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:101) at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:31658) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:268