Created 08-23-2016 07:51 AM
My cluster have registered hosts but do not added on cluster.
It means that,
I was installed ambari-agent on some hosts.
This hosts registered to ambari-server by changing server hostname on amber-agent.ini
But, I didn't registered on cluster using ambari web UI.
Some times later, when i was checked ambari-server log, ambari server still check this host's heartbeat.
"WARN [ambari-hearbeat-monitor] HeartbeatMonitor:154 - Heartbeat lost from host hostname"
how can i remove this hosts?
Created 08-23-2016 11:37 AM
You have a "hoststate" and/or "hosts" table in ambari database which might have an entry of that hostsname which will give you that WARN [ambari-hearbeat-monitor] HeartbeatMonitor:154 - Heartbeat lost from host hostname" on every ambari-server restart is performed as it checks those tables. You might want to have a check if the entry exists then remove those entries by stopping the ambari-server and stopping the ambari-agent of those host which gives you that warning.
Because if the ambari-agent is in running state then you will see the same issue again.
Created 08-23-2016 07:58 AM
Do you see these unwanted hosts under 'hosts' tab in ambari?
Then click on that particular host's page and goto host details page. From the 'hosts action' dropdown, delete the host.
Attaching the screenshots.
Created 08-23-2016 08:11 AM
I don't add on Ambari web ui. So, I can't see anywhere.
But, ambari-server still check heartbeat.
When i try to add host, on Confirm hosts tab, i can see this hosts on 'Other Registered Hosts' menu.
This menu say that, "These are the hosts that have registered with the server, but do not appear in the list of hosts that you are adding."
Created 08-23-2016 08:29 AM
To remove the host from the cluster follow below steps -
1. Stop the ambari-agent on the node. you want to remove.
2. Remove the ambari-agent packages from that node using $yum remove ambari-agent
3. On safer side you can run $hdfs dfsadmin -refreshNamenodes command to refresh nodes on cluster
4. Confirm using $hdfs dfsadmin -report the host is no more part of cluster.
Created 08-23-2016 09:04 AM
"hdfs dfsadmin -report" command is not shown registered but don't added hosts.
refreshNamenodes datanodehost:port | For the given datanode, reloads the configuration files, stops serving the removed block-pools and starts serving new block-pools. |
refreshNodes | Re-read the hosts and exclude files to update the set of Datanodes that are allowed to connect to the Namenode and those that should be decommissioned or recommissioned. |
I think refreshNamenodes command is not correct in my problem, so i try to refreshNodes command, but it doesn't works. Ambari-server still write heartbeat lost log.
Created 08-23-2016 09:14 AM
I was found this hosts ambari.hosts and ambari.hoststate on postgre database.
This host's health_status is UNKNOWN.
I want to remove this hosts clearly.
Created 08-23-2016 09:47 AM
Ideally on every restart of ambari agent it sends a registration request to the ambari server along with it's latest (Hostname+IPAddress). That is then stored in the "hosts" table of ambari DB.
You can check the hostnames that are registered using the curl command as following:
curl -i -H "X-Requested-By: ambari"-u admin:admin -X GET http://localhost:8080/api/v1/hosts
So if you do not want to incorrect agent to be registered then uninstall that agent and do not restart it back.
Created 08-24-2016 12:45 AM
This curl command shown registered hosts. thx!
I try to below command, it works.
curl -i -H "X-Requested-By: ambari"-u admin:admin -X DELETE http://localhost:8080/api/v1/hosts/hotname
Created 02-22-2018 08:54 AM
Above curl command worked in my case. Thanks.
Created 08-23-2016 11:37 AM
You have a "hoststate" and/or "hosts" table in ambari database which might have an entry of that hostsname which will give you that WARN [ambari-hearbeat-monitor] HeartbeatMonitor:154 - Heartbeat lost from host hostname" on every ambari-server restart is performed as it checks those tables. You might want to have a check if the entry exists then remove those entries by stopping the ambari-server and stopping the ambari-agent of those host which gives you that warning.
Because if the ambari-agent is in running state then you will see the same issue again.