datanodes are running but only one node is showing live out of 3.

[#1]

Hi All,

I've setup cluster newly and installed datanode in slave servers. After starting up HDFS cluster fully, I see only one datanode is live out of 3 nodes. I checked cluster ID, datanode Uuid, pool ID are same in each datanode. Please help to resolve this issue.

Thanks in advance.

[#2]

@Saravana V,

Can you check the logs under /var/log/hadoop/hdfs/ folder to see if there are any errors found in the datanode logs. It would be great if you can attach the logs to investigate.

.

-Aditya

[#3]

@Aditya Sirna

Please see the log below.

$ tail -f hadoop-hdfs-datanode-HKLPADBID04.log

2018-10-05 21:50:01,820 INFO datanode.DataNode (BPServiceActor.java:register(734)) - Block pool Block pool BP-1265401458-192.168.22.18-1538217251820 (Datanode Uuid 9ba5d3f6-bd76-436f-a52c-bb4bc6c3d970) service to hklpadbid02.hk.standardchartered.com/192.168.22.18:8020 successfully registered with NN

2018-10-05 21:50:01,820 INFO block.BlockTokenSecretManager (BlockTokenSecretManager.java:addKeys(193)) - Setting block keys 2018-10-05 21:50:04,718 INFO datanode.DataNode (BPOfferService.java:processCommandFromActor(609)) - DatanodeCommand action : DNA_REGISTER from hklpadbid03.hk.standardchartered.com/192.168.22.19:8020 with standby state

2018-10-05 21:50:04,718 INFO datanode.DataNode (BPServiceActor.java:register(715)) - Block pool BP-1265401458-192.168.22.18-1538217251820 (Datanode Uuid 9ba5d3f6-bd76-436f-a52c-bb4bc6c3d970) service to hklpadbid03.hk.standardchartered.com/192.168.22.19:8020 beginning handshake with NN

2018-10-05 21:50:04,719 INFO datanode.DataNode (BPServiceActor.java:register(734)) - Block pool Block pool BP-1265401458-192.168.22.18-1538217251820 (Datanode Uuid 9ba5d3f6-bd76-436f-a52c-bb4bc6c3d970) service to hklpadbid03.hk.standardchartered.com/192.168.22.19:8020 successfully registered with NN

2018-10-05 21:50:04,719 INFO block.BlockTokenSecretManager (BlockTokenSecretManager.java:addKeys(193)) - Setting block keys 2018-10-05 21:50:04,821 INFO datanode.DataNode (BPOfferService.java:processCommandFromActor(609)) - DatanodeCommand action : DNA_REGISTER from hklpadbid02.hk.standardchartered.com/192.168.22.18:8020 with active state

2018-10-05 21:50:04,821 INFO datanode.DataNode (BPServiceActor.java:register(715)) - Block pool BP-1265401458-192.168.22.18-1538217251820 (Datanode Uuid 9ba5d3f6-bd76-436f-a52c-bb4bc6c3d970) service to hklpadbid02.hk.standardchartered.com/192.168.22.18:8020 beginning handshake with NN

2018-10-05 21:50:04,822 INFO datanode.DataNode (BPServiceActor.java:register(734)) - Block pool Block pool BP-1265401458-192.168.22.18-1538217251820 (Datanode Uuid 9ba5d3f6-bd76-436f-a52c-bb4bc6c3d970) service to hklpadbid02.hk.standardchartered.com/192.168.22.18:8020 successfully registered with NN

2018-10-05 21:50:04,822 INFO block.BlockTokenSecretManager (BlockTokenSecretManager.java:addKeys(193)) - Setting block keys 2018-10-05 21:50:07,720 INFO datanode.DataNode (BPOfferService.java:processCommandFromActor(609)) - DatanodeCommand action : DNA_REGISTER from hklpadbid03.hk.standardchartered.com/192.168.22.19:8020 with standby state

2018-10-05 21:50:07,720 INFO datanode.DataNode (BPServiceActor.java:register(715)) - Block pool BP-1265401458-192.168.22.18-1538217251820 (Datanode Uuid 9ba5d3f6-bd76-436f-a52c-bb4bc6c3d970) service to hklpadbid03.hk.standardchartered.com/192.168.22.19:8020 beginning handshake with NN

2018-10-05 21:50:07,721 INFO datanode.DataNode (BPServiceActor.java:register(734)) - Block pool Block pool BP-1265401458-192.168.22.18-1538217251820 (Datanode Uuid 9ba5d3f6-bd76-436f-a52c-bb4bc6c3d970) service to hklpadbid03.hk.standardchartered.com/192.168.22.19:8020 successfully registered with NN

2018-10-05 21:50:07,721 INFO block.BlockTokenSecretManager (BlockTokenSecretManager.java:addKeys(193)) - Setting block keys 2018-10-05 21:50:07,823 INFO datanode.DataNode (BPOfferService.java:processCommandFromActor(609)) - DatanodeCommand action : DNA_REGISTER from hklpadbid02.hk.standardchartered.com/192.168.22.18:8020 with active state

2018-10-05 21:50:07,823 INFO datanode.DataNode (BPServiceActor.java:register(715)) - Block pool BP-1265401458-192.168.22.18-1538217251820 (Datanode Uuid 9ba5d3f6-bd76-436f-a52c-bb4bc6c3d970) service to hklpadbid02.hk.standardchartered.com/192.168.22.18:8020 beginning handshake with NN

2018-10-05 21:50:07,824 INFO datanode.DataNode (BPServiceActor.java:register(734)) - Block pool Block pool BP-1265401458-192.168.22.18-1538217251820 (Datanode Uuid 9ba5d3f6-bd76-436f-a52c-bb4bc6c3d970) service to hklpadbid02.hk.standardchartered.com/192.168.22.18:8020 successfully registered with NN

2018-10-05 21:50:07,824 INFO block.BlockTokenSecretManager (BlockTokenSecretManager.java:addKeys(193)) - Setting block keys

[#4]

@Saravana V,

It doesn't have any error logs. Can you see the file if you have any errors

[#5]

@Saravana V These are all INFO messages, makes me believe that since additional datanodes have been added, rebalance is taking place. See if you can find any exception in the HDFS logs.