Support Questions

Find answers, ask questions, and share your expertise

Deleted Hosts still appear in Grafana

avatar
Super Collaborator

Hello all

I've noticed that in the Grafana UI when i filter for specific servers i can still see in the list Hosts that are no longer a part of the cluster (Datanodes that were deleted completely from the cluster).
How can i remove them from Grafana UI ?

Adi

1 ACCEPTED SOLUTION

avatar
Master Mentor

@Adi Jabkowsky

Grafana collects information from Ambari Metrics collector.

Ambari metrics collecotr stores hisgtorical metrics data. So your Old Host data will still be there.

If you want to verify your host removal changes then click at the top right corner of the Grafana UI "Last 6 Hours" data to change the time after you deleted the Host.

Other options may be to set the TTL value for the host level metrics (timeline.metrics.host.aggregator.daily.ttl) to a small value so that older data from the specified date is not maintained inside the AMS database (However this approach will cause deletion of other host metrics from the specified date)

Please see this link for more information on TTL: https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-operations/content/ams_aggregated...

View solution in original post

4 REPLIES 4

avatar
Master Mentor

@Adi Jabkowsky

Grafana collects information from Ambari Metrics collector.

Ambari metrics collecotr stores hisgtorical metrics data. So your Old Host data will still be there.

If you want to verify your host removal changes then click at the top right corner of the Grafana UI "Last 6 Hours" data to change the time after you deleted the Host.

Other options may be to set the TTL value for the host level metrics (timeline.metrics.host.aggregator.daily.ttl) to a small value so that older data from the specified date is not maintained inside the AMS database (However this approach will cause deletion of other host metrics from the specified date)

Please see this link for more information on TTL: https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-operations/content/ams_aggregated...

avatar
Super Collaborator

@Jay Kumar SenSharma

Thank you for your promptly response !
I was refering to hosts that were deleted months ago that are still there...
So it seems that Grafana isn't aware to the change....

avatar
Master Mentor

@Adi Jabkowsky

Grafana pulls all historical metrics related data from AMS collector, So your AMS might be having the old host information.

Please try accessing the following URL of AMS collector to see how many hosts are there and to verify the AMS database still has Old host info:

http://<ams-host>:6188/ws/v1/timeline/metrics/hosts

avatar
Super Collaborator

@Jay Kumar SenSharma

The old hosts are still in the AMS metrics. I will take care of the TTL in AMS as per your recommendations.
Thank you !