Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Is this a bug of ambari ? this occur when delete a host.

avatar
Explorer

18383-1212.png

1 ACCEPTED SOLUTION

avatar
Master Mentor

@jack jack

Surely 8/7 is not right.

But i was thinking that When you deleted a host from ambari cluster the host count became 7 , But before deleting the host did you Stop the DataNode running on that host? Else from NameNode perspective the Running DataNodes will be still 8 because that DataNode is still reporting to the NameNode.

.

Because DataNodes are reports to the NameNode, So you should check the NameNode UI to findout the exact number of DataNode.

http://$NAMENODE:50070/dfshealth.html#tab-overview  Find the "Live DataNodes"

.

Ambari simply grabs the LiveDataNode count from the NameNode JMX

http://$NAMENODE:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystemState

.

So please check the Deleted Host if the DataNode process is still running there? If yes then stop it.

# ps -ef | grep DateNode
# echo `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`
# ps -ef | grep  `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`

######## If the PID is live and running then kill it. ######
# kill -9  `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`

.

View solution in original post

6 REPLIES 6

avatar
Explorer

avatar
Master Mentor

@jack jack

Surely 8/7 is not right.

But i was thinking that When you deleted a host from ambari cluster the host count became 7 , But before deleting the host did you Stop the DataNode running on that host? Else from NameNode perspective the Running DataNodes will be still 8 because that DataNode is still reporting to the NameNode.

.

Because DataNodes are reports to the NameNode, So you should check the NameNode UI to findout the exact number of DataNode.

http://$NAMENODE:50070/dfshealth.html#tab-overview  Find the "Live DataNodes"

.

Ambari simply grabs the LiveDataNode count from the NameNode JMX

http://$NAMENODE:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystemState

.

So please check the Deleted Host if the DataNode process is still running there? If yes then stop it.

# ps -ef | grep DateNode
# echo `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`
# ps -ef | grep  `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`

######## If the PID is live and running then kill it. ######
# kill -9  `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`

.

avatar
Explorer

the datanode is stopped before delete the host. but when I see the DataNodes Live through jmx, the datanode is still live. How to solve it ?

avatar
Master Mentor

@jack jack

If the NameNode is returning incorrect data then From Ambari side we can not do much.

Please check the NameNode UI to see if you are finding the problematic DataNode listed there?

http://$NAMENODE_HOST:50070/dfshealth.html#tab-datanode

.

Please check if you see the DataNode name listed in the above URL?

Try restarting the NameNode and then see if it fixes the stale datanode list.

.

avatar
Explorer

the NameNode returning is still 8, can we change the dashboard get data from HDFS Summary.Because when I delete host the HDFS summary area display correctly

avatar
Explorer

20469-9.png

this label is sensitive and correct