Deleted Hosts still appear in Grafana

[#1]

Hello all

I've noticed that in the Grafana UI when i filter for specific servers i can still see in the list Hosts that are no longer a part of the cluster (Datanodes that were deleted completely from the cluster).
How can i remove them from Grafana UI ?

Adi

[#2 · Accepted solution]

@Adi Jabkowsky

Grafana collects information from Ambari Metrics collector.

Ambari metrics collecotr stores hisgtorical metrics data. So your Old Host data will still be there.

If you want to verify your host removal changes then click at the top right corner of the Grafana UI "Last 6 Hours" data to change the time after you deleted the Host.

Other options may be to set the TTL value for the host level metrics (timeline.metrics.host.aggregator.daily.ttl) to a small value so that older data from the specified date is not maintained inside the AMS database (However this approach will cause deletion of other host metrics from the specified date)

Please see this link for more information on TTL: https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-operations/content/ams_aggregated...

[#3]

@Jay Kumar SenSharma

Thank you for your promptly response !
I was refering to hosts that were deleted months ago that are still there...
So it seems that Grafana isn't aware to the change....

[#4]

@Adi Jabkowsky

Grafana pulls all historical metrics related data from AMS collector, So your AMS might be having the old host information.

Please try accessing the following URL of AMS collector to see how many hosts are there and to verify the AMS database still has Old host info:

http://<ams-host>:6188/ws/v1/timeline/metrics/hosts

[#5]

@Jay Kumar SenSharma

The old hosts are still in the AMS metrics. I will take care of the TTL in AMS as per your recommendations.
Thank you !