Support Questions

Find answers, ask questions, and share your expertise

Is this a bug of ambari ? this occur when delete a host.

Explorer

18383-1212.png

1 ACCEPTED SOLUTION

Super Mentor

@jack jack

Surely 8/7 is not right.

But i was thinking that When you deleted a host from ambari cluster the host count became 7 , But before deleting the host did you Stop the DataNode running on that host? Else from NameNode perspective the Running DataNodes will be still 8 because that DataNode is still reporting to the NameNode.

.

Because DataNodes are reports to the NameNode, So you should check the NameNode UI to findout the exact number of DataNode.

http://$NAMENODE:50070/dfshealth.html#tab-overview  Find the "Live DataNodes"

.

Ambari simply grabs the LiveDataNode count from the NameNode JMX

http://$NAMENODE:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystemState

.

So please check the Deleted Host if the DataNode process is still running there? If yes then stop it.

# ps -ef | grep DateNode
# echo `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`
# ps -ef | grep  `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`

######## If the PID is live and running then kill it. ######
# kill -9  `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`

.

View solution in original post

6 REPLIES 6

Explorer

Super Mentor

@jack jack

Surely 8/7 is not right.

But i was thinking that When you deleted a host from ambari cluster the host count became 7 , But before deleting the host did you Stop the DataNode running on that host? Else from NameNode perspective the Running DataNodes will be still 8 because that DataNode is still reporting to the NameNode.

.

Because DataNodes are reports to the NameNode, So you should check the NameNode UI to findout the exact number of DataNode.

http://$NAMENODE:50070/dfshealth.html#tab-overview  Find the "Live DataNodes"

.

Ambari simply grabs the LiveDataNode count from the NameNode JMX

http://$NAMENODE:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystemState

.

So please check the Deleted Host if the DataNode process is still running there? If yes then stop it.

# ps -ef | grep DateNode
# echo `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`
# ps -ef | grep  `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`

######## If the PID is live and running then kill it. ######
# kill -9  `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`

.

Explorer

the datanode is stopped before delete the host. but when I see the DataNodes Live through jmx, the datanode is still live. How to solve it ?

Super Mentor

@jack jack

If the NameNode is returning incorrect data then From Ambari side we can not do much.

Please check the NameNode UI to see if you are finding the problematic DataNode listed there?

http://$NAMENODE_HOST:50070/dfshealth.html#tab-datanode

.

Please check if you see the DataNode name listed in the above URL?

Try restarting the NameNode and then see if it fixes the stale datanode list.

.

Explorer

the NameNode returning is still 8, can we change the dashboard get data from HDFS Summary.Because when I delete host the HDFS summary area display correctly

Explorer

20469-9.png

this label is sensitive and correct

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.