Support Questions
Find answers, ask questions, and share your expertise

datanodes are running but only one node is showing live out of 3.

Explorer

Hi All,

I've setup cluster newly and installed datanode in slave servers. After starting up HDFS cluster fully, I see only one datanode is live out of 3 nodes. I checked cluster ID, datanode Uuid, pool ID are same in each datanode. Please help to resolve this issue.

Thanks in advance.

4 REPLIES 4

@Saravana V,

Can you check the logs under /var/log/hadoop/hdfs/ folder to see if there are any errors found in the datanode logs. It would be great if you can attach the logs to investigate.

.

-Aditya

Explorer

@Aditya Sirna

Please see the log below.

$ tail -f hadoop-hdfs-datanode-HKLPADBID04.log

2018-10-05 21:50:01,820 INFO datanode.DataNode (BPServiceActor.java:register(734)) - Block pool Block pool BP-1265401458-192.168.22.18-1538217251820 (Datanode Uuid 9ba5d3f6-bd76-436f-a52c-bb4bc6c3d970) service to hklpadbid02.hk.standardchartered.com/192.168.22.18:8020 successfully registered with NN

2018-10-05 21:50:01,820 INFO block.BlockTokenSecretManager (BlockTokenSecretManager.java:addKeys(193)) - Setting block keys 2018-10-05 21:50:04,718 INFO datanode.DataNode (BPOfferService.java:processCommandFromActor(609)) - DatanodeCommand action : DNA_REGISTER from hklpadbid03.hk.standardchartered.com/192.168.22.19:8020 with standby state

2018-10-05 21:50:04,718 INFO datanode.DataNode (BPServiceActor.java:register(715)) - Block pool BP-1265401458-192.168.22.18-1538217251820 (Datanode Uuid 9ba5d3f6-bd76-436f-a52c-bb4bc6c3d970) service to hklpadbid03.hk.standardchartered.com/192.168.22.19:8020 beginning handshake with NN

2018-10-05 21:50:04,719 INFO datanode.DataNode (BPServiceActor.java:register(734)) - Block pool Block pool BP-1265401458-192.168.22.18-1538217251820 (Datanode Uuid 9ba5d3f6-bd76-436f-a52c-bb4bc6c3d970) service to hklpadbid03.hk.standardchartered.com/192.168.22.19:8020 successfully registered with NN

2018-10-05 21:50:04,719 INFO block.BlockTokenSecretManager (BlockTokenSecretManager.java:addKeys(193)) - Setting block keys 2018-10-05 21:50:04,821 INFO datanode.DataNode (BPOfferService.java:processCommandFromActor(609)) - DatanodeCommand action : DNA_REGISTER from hklpadbid02.hk.standardchartered.com/192.168.22.18:8020 with active state

2018-10-05 21:50:04,821 INFO datanode.DataNode (BPServiceActor.java:register(715)) - Block pool BP-1265401458-192.168.22.18-1538217251820 (Datanode Uuid 9ba5d3f6-bd76-436f-a52c-bb4bc6c3d970) service to hklpadbid02.hk.standardchartered.com/192.168.22.18:8020 beginning handshake with NN

2018-10-05 21:50:04,822 INFO datanode.DataNode (BPServiceActor.java:register(734)) - Block pool Block pool BP-1265401458-192.168.22.18-1538217251820 (Datanode Uuid 9ba5d3f6-bd76-436f-a52c-bb4bc6c3d970) service to hklpadbid02.hk.standardchartered.com/192.168.22.18:8020 successfully registered with NN

2018-10-05 21:50:04,822 INFO block.BlockTokenSecretManager (BlockTokenSecretManager.java:addKeys(193)) - Setting block keys 2018-10-05 21:50:07,720 INFO datanode.DataNode (BPOfferService.java:processCommandFromActor(609)) - DatanodeCommand action : DNA_REGISTER from hklpadbid03.hk.standardchartered.com/192.168.22.19:8020 with standby state

2018-10-05 21:50:07,720 INFO datanode.DataNode (BPServiceActor.java:register(715)) - Block pool BP-1265401458-192.168.22.18-1538217251820 (Datanode Uuid 9ba5d3f6-bd76-436f-a52c-bb4bc6c3d970) service to hklpadbid03.hk.standardchartered.com/192.168.22.19:8020 beginning handshake with NN

2018-10-05 21:50:07,721 INFO datanode.DataNode (BPServiceActor.java:register(734)) - Block pool Block pool BP-1265401458-192.168.22.18-1538217251820 (Datanode Uuid 9ba5d3f6-bd76-436f-a52c-bb4bc6c3d970) service to hklpadbid03.hk.standardchartered.com/192.168.22.19:8020 successfully registered with NN

2018-10-05 21:50:07,721 INFO block.BlockTokenSecretManager (BlockTokenSecretManager.java:addKeys(193)) - Setting block keys 2018-10-05 21:50:07,823 INFO datanode.DataNode (BPOfferService.java:processCommandFromActor(609)) - DatanodeCommand action : DNA_REGISTER from hklpadbid02.hk.standardchartered.com/192.168.22.18:8020 with active state

2018-10-05 21:50:07,823 INFO datanode.DataNode (BPServiceActor.java:register(715)) - Block pool BP-1265401458-192.168.22.18-1538217251820 (Datanode Uuid 9ba5d3f6-bd76-436f-a52c-bb4bc6c3d970) service to hklpadbid02.hk.standardchartered.com/192.168.22.18:8020 beginning handshake with NN

2018-10-05 21:50:07,824 INFO datanode.DataNode (BPServiceActor.java:register(734)) - Block pool Block pool BP-1265401458-192.168.22.18-1538217251820 (Datanode Uuid 9ba5d3f6-bd76-436f-a52c-bb4bc6c3d970) service to hklpadbid02.hk.standardchartered.com/192.168.22.18:8020 successfully registered with NN

2018-10-05 21:50:07,824 INFO block.BlockTokenSecretManager (BlockTokenSecretManager.java:addKeys(193)) - Setting block keys

@Saravana V,

It doesn't have any error logs. Can you see the file if you have any errors

Guru

@Saravana V These are all INFO messages, makes me believe that since additional datanodes have been added, rebalance is taking place. See if you can find any exception in the HDFS logs.

; ;