Question on Ambari Metrics

[#1]

Hi,

We have an org-wide monitoring and alerting system that is being used. We now want to integrate HDP related metrics as well into this system which works on scanning the log files with regular expressions and alerting in case of any threshold breaches.

I have the following questions regarding this scenario:

1. Whether it is sufficient for the monitoring system to just scan the ambari-alerts.log alone to get all HDP related metrics? Or is there a need to monitor the individual component level log files as well? For example, to monitor Storm, Kafka, Solr etc, should each individual log files be monitored?
2. What additional details will the individual component level logs provide in comparison to the ambari-alerts.log?
3. While Grafana has dashboards for many metrics, Ambari Alerts are very few in comparison. Is there an easy way to configure all the Grafana metrics to be considered for alerting?

Thanks in advance.

[#2]

@Greenhorn Techie

1. I don't think its good idea to scan the log files for HDP related metrics. ambari-alerts.log does not have any metrics. AMS provides APIs to read the metrics and you can try and use that - you can find documentation at https://cwiki-test.apache.org/confluence/display/AMBARI/Metrics+Collector+API+Specification

2. ambari-alert.log is only for debugging purpose - if you would like to find out the alerts generated then there is a notification feature which you can consider using that.

3. Currently Grafana does not provide an option (or even Ambari) to configure the alerts for metrics.

Note: Please mark this as correct answer if you satiesfied

[#3]

@Greenhorn Techie As I mentioned - Ambari is capable of sending alert notifications- you can consider using that if your system is capable of processing SNMP/EMAIL notifications.

[#4]

@amarnath reddy pappu our organisation requirement is to use enterprise wide tool instead of Ambari and hence that option is ruled out altogether.

[#5]

@Greenhorn Techie

Ambari is capable of taking the custom script as input - so you can write your own custom script and within that you decide what ever way you wanted. you can follow the document at https://risdenk.github.io/2018/03/25/apache-ambari-custom-alert-dispatch-script.html

[#6]

@Greenhorn Techie If you are happy with the explanation/solution then please accept the correct answer so that it will help others.

[#7]

Thanks @amarnath reddy pappu for your response. It provides some food for thought to further explore and better our understanding. We intend to use the external system mainly for alerting purposes so that the support team can act quickly in case any action is required. I was thinking of using the ambari-alerts.log and other log files for only alerting purposes.

As the present alerting mechanism is only within Ambari, and our Ops team want to use their existing setup, wondering what is the best possible alternative if scanning the log files is not the way to go?

[#8]

Very nice and interesting
https://www.welookups.com