Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Is this a bug of ambari ? this occur when delete a host.

Solved Go to solution

Is this a bug of ambari ? this occur when delete a host.

New Contributor

18383-1212.png

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Is this a bug of ambari ? this occur when delete a host.

Super Mentor

@jack jack

Surely 8/7 is not right.

But i was thinking that When you deleted a host from ambari cluster the host count became 7 , But before deleting the host did you Stop the DataNode running on that host? Else from NameNode perspective the Running DataNodes will be still 8 because that DataNode is still reporting to the NameNode.

.

Because DataNodes are reports to the NameNode, So you should check the NameNode UI to findout the exact number of DataNode.

http://$NAMENODE:50070/dfshealth.html#tab-overview  Find the "Live DataNodes"

.

Ambari simply grabs the LiveDataNode count from the NameNode JMX

http://$NAMENODE:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystemState

.

So please check the Deleted Host if the DataNode process is still running there? If yes then stop it.

# ps -ef | grep DateNode
# echo `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`
# ps -ef | grep  `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`

######## If the PID is live and running then kill it. ######
# kill -9  `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`

.

6 REPLIES 6

Re: Is this a bug of ambari ? this occur when delete a host.

New Contributor

Re: Is this a bug of ambari ? this occur when delete a host.

Super Mentor

@jack jack

Surely 8/7 is not right.

But i was thinking that When you deleted a host from ambari cluster the host count became 7 , But before deleting the host did you Stop the DataNode running on that host? Else from NameNode perspective the Running DataNodes will be still 8 because that DataNode is still reporting to the NameNode.

.

Because DataNodes are reports to the NameNode, So you should check the NameNode UI to findout the exact number of DataNode.

http://$NAMENODE:50070/dfshealth.html#tab-overview  Find the "Live DataNodes"

.

Ambari simply grabs the LiveDataNode count from the NameNode JMX

http://$NAMENODE:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystemState

.

So please check the Deleted Host if the DataNode process is still running there? If yes then stop it.

# ps -ef | grep DateNode
# echo `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`
# ps -ef | grep  `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`

######## If the PID is live and running then kill it. ######
# kill -9  `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`

.

Re: Is this a bug of ambari ? this occur when delete a host.

New Contributor

the datanode is stopped before delete the host. but when I see the DataNodes Live through jmx, the datanode is still live. How to solve it ?

Re: Is this a bug of ambari ? this occur when delete a host.

Super Mentor

@jack jack

If the NameNode is returning incorrect data then From Ambari side we can not do much.

Please check the NameNode UI to see if you are finding the problematic DataNode listed there?

http://$NAMENODE_HOST:50070/dfshealth.html#tab-datanode

.

Please check if you see the DataNode name listed in the above URL?

Try restarting the NameNode and then see if it fixes the stale datanode list.

.

Re: Is this a bug of ambari ? this occur when delete a host.

New Contributor

the NameNode returning is still 8, can we change the dashboard get data from HDFS Summary.Because when I delete host the HDFS summary area display correctly

Re: Is this a bug of ambari ? this occur when delete a host.

New Contributor

20469-9.png

this label is sensitive and correct

Don't have an account?
Coming from Hortonworks? Activate your account here