question Ambari dashboard - "DataNodes Live" shows 1 live, 0 dead, 0 decompressioning in Archives of Support Questions (Read Only)

Ambari dashboard - "DataNodes Live" shows 1 live, 0 dead, 0 decompressioning

mikeskali — Sun, 08 May 2016 21:15:09 GMT

We are using HDP 2.3, Ambari 2.1.2

We have connected 3 datanodes to the cluster (slave1, slave2, slave3)

Ambari server is running on a distinct server.

NameNode is running on slave1.

Dashboard shows:

DataNodes Live 1/3 (0 dead, 0 decompressioning)

NodeManagers Live 3/3

I can access all three data nodes from ambari "Hosts". I can restart the services on these data nodes.

NameNode UI shows 1 live data node (slave1 - same one with NameNode installed on).

Re: Ambari dashboard - "DataNodes Live" shows 1 live, 0 dead, 0 decompressioning

KuldeepK — Sun, 08 May 2016 21:36:28 GMT

@michael sklyar

Can you please run below commands as hdfs user and share output?

#Command 1:

hadoop dfsadmin -report

#Command 2:

hadoop dfsadmin -refreshNodes

After running above command, please check your Namenode UI to see if live nodes are 3/3. If possible, please share screenshot of NN UI.

Re: Ambari dashboard - "DataNodes Live" shows 1 live, 0 dead, 0 decompressioning

mikeskali — Mon, 19 Aug 2019 08:48:37 GMT

1. report:

Configured Capacity: 51636727808 (48.09 GB)
Present Capacity: 28807827456 (26.83 GB)
DFS Remaining: 28329484288 (26.38 GB)
DFS Used: 478343168 (456.18 MB)
DFS Used%: 1.66%
Under replicated blocks: 1023
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0


-------------------------------------------------
Live datanodes (1):


Name: 192.168.41.10:50010 (stream-srv01.streaming.rnd.qa)
Hostname: stream-srv01.streaming.rnd.qa
Decommission Status : Normal
Configured Capacity: 51636727808 (48.09 GB)
DFS Used: 478343168 (456.18 MB)
Non DFS Used: 22828900352 (21.26 GB)
DFS Remaining: 28329484288 (26.38 GB)
DFS Used%: 0.93%
DFS Remaining%: 54.86%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Sun May 08 20:43:40 IDT 2016

2. refresh command output: Refresh nodes successful

3. NameNode UI shows 1 live data node (the one with namenode installed on):

Re: Ambari dashboard - "DataNodes Live" shows 1 live, 0 dead, 0 decompressioning

PranayV — Mon, 09 May 2016 01:05:59 GMT

Is the datanode service within each slave node appearing started in Ambari?. Also on any one of the slave node that is not working check for errors on /var/log/hadoop/hdfs/<datanodelog> file?

Also on slave 1 check namenode log file to see if the datanodes are trying to heartbeat to namenode.

Regards

Pranay Vyas

Re: Ambari dashboard - "DataNodes Live" shows 1 live, 0 dead, 0 decompressioning

sshimpi — Mon, 09 May 2016 01:53:19 GMT

@michael sklyar

It might be the case that datanode process is kind of in hung state and and is not responding the status update.

Did you tried restarting the datanode service?

can you share namenode and datanode logs ?

Re: Ambari dashboard - "DataNodes Live" shows 1 live, 0 dead, 0 decompressioning

mikeskali — Mon, 09 May 2016 16:05:36 GMT

Thank you!

in datanode log we got the following error:

datanode.DataNode (BPServiceActor.java:run(828)) - Initialization failed for Block pool BP-1743137494-192.168.41.10-1459773600716 (Datanode Uuid 41b4525d-b168-496d-a985-a2c2b5e889c1) service to slave1/192.168.41.10:8020 Datanode denied communication with namenode because hostname cannot be resolved (ip=192.168.41.11, hostname=192.168.41.11): DatanodeRegistration(0.0.0.0:50010, datanodeUuid=41b4525d-b168-496d-a985-a2c2b5e889c1, infoPort=50075, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-56;cid=CID-655f2b29-eb31-4c17-81b6-b507e7f6cb5f;nsid=1493722973;c=0)

The problem was in the reverse DNS resolving (a check performed by the namenode).

This article was very helpful: why-datanode-is-denied-communication-with-namenode

Re: Ambari dashboard - "DataNodes Live" shows 1 live, 0 dead, 0 decompressioning

dpnctl — Wed, 07 Dec 2016 22:50:55 GMT

I deployed a clusteron vmware and becuae I was having issues to add additional node, I have cloned a VM, changed hostname and up and added the datanode manually. As I was seeing just one datanode on dashboard, I tried making all changes as suggested here which actually didnt work. As soon as I shutdown the an active datanode , the second one started to show up. Any clues what was happening in back end?

Re: Ambari dashboard - "DataNodes Live" shows 1 live, 0 dead, 0 decompressioning

dpnctl — Thu, 08 Dec 2016 20:19:39 GMT

my assumption was correct the datanodes (prbly everynode) will have a uuid which was same and hence this issue, i removed the install software, diectories and files, then reregisterd which worked fine later