Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Datanode connect to standby namenode and keep upgrading block files after restart

Highlighted

Datanode connect to standby namenode and keep upgrading block files after restart

New Contributor

Duaring major upgrading our HDP to 3.1 version, we notice some datanodes are connecting to standby namenode. So the active namenode raises missing block warnings.

We tried to restart datanodes to fix this, however, more confusing things happened. The restart datanode began to upgrading its block files again (which tooks tens of minutes).

We have no idea how to fix this problem. Any suggestion is appreciate!

Following is node status on standby and active masters:

106533-1550686790736.png


And some log from datanode and namenode

DataNode:

2019-02-21 05:40:20,242 WARN datanode.DataNode (BPServiceActor.java:offerService(721)) - Block pool BP-1766603066-IP-1521601582138 (Datanode Uuid 9e0b7314-b138-486a-a4e5-0fe9f0328be5) service to master/IP:8020 is shutting down
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.UnregisteredNodeException): Unregistered server: DatanodeRegistration(IP:50010, datanodeUuid=9e0b7314-b138-486a-a4e5-0fe9f0328be5, infoPort=50075, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-57;cid=CID-cb189bc7-726d-48a3-916a-5c99ad2ff653;nsid=798722167;c=1550662791172)

NameNode:
2019-02-21 06:16:07,841 INFO ipc.Server (Server.java:logException(2726)) - IPC Server handler 26 on 8020, call Call#4 Retry#0 org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol.sendHeartbeat from IP:35198
org.apache.hadoop.hdfs.protocol.UnregisteredNodeException: Unregistered server: DatanodeRegistration(IP:50010, datanodeUuid=b9e401bf-53c5-4907-8781-ce5d465e9e69, infoPort=50075, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-57;cid=CID-cb189bc7-726d-48a3-916a-5c99ad2ff653;nsid=798722167;c=0)