WARN [ambari-hearbeat-monitor] HeartbeatMonitor:154 - Heartbeat lost from host

[#1]

I have a multinode cluster and Ive been adding and removing hosts to the cluster. Ambaris database is MySql.

I have removed a few datanodes and now I see in ambari-server.log the following warning:

WARN [ambari-hearbeat-monitor] HeartbeatMonitor:154 - Heartbeat lost from host <hostname>

This is showing for hostnames that I have removed from the cluster.

What I have done for one hostname and it seemed to work:

I went to mysql database ambari_db and ran these two commands:

delete from hoststate where host_id = 'hostname_id'; delete from hosts where host_name = 'hostname'; commit; flush privileges

restarting ambari-server

The WARN for that particular hostname is not showing up again.

Is this the best practice to get rid of the hostnames that were removed from the cluster?

Thanks in advance!

[#2]

@marko

You need to use Ambari API to delete hosts. The best practice is to use this. you also need to remove the ambari agent if you choose to delete host. You can also run host clean up script afterwards to remove any residual files. Going directly to database and removing hosts is not recommended. Here is also some info.

[#3]

@marko

I believe you leveraged ambari to delete the host. Ambari should take care of syncing the node removal. You should not have to run delete from hoststate where host_id = 'hostname_id'; delete from hosts where host_name = 'hostname'; commit; flush privileges

Check ambari logs for more details on what happened during deleting the nodes.

[#4 · Accepted solution]

@Artem Ervits @Neeraj Sabharwal

I followed the following website when removing the hosts: https://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_Ambari_Users_Guide/content/_how_to_delet...

Prior to that, I shut down all services on the to-be-deleted hosts, shut down the ambari-agent.

Those hosts do not exist anymore so the clean up script cannot be accessed. I will use the script next time, at the moment, I would like to see if there is anything else I need to take care of, except deleting the hosts in the database.

Maybe Hortonworks people can update the procedure on the above mentioned website?