question Re: How to view Alert History via Ambari ? in Support Questions

How to view Alert History via Ambari ?

Jagatheeshr — Wed, 07 Oct 2015 20:12:10 GMT

I want to view how many times Kafka died in the last 10 days . Is there any ways to view it from Ambari ?

Re: How to view Alert History via Ambari ?

bwilson — Wed, 07 Oct 2015 23:47:55 GMT

I am not aware of a historical view into this information via the Ambari UI but you can certainly grab this information via the backing Ambari database. Using the default installation you can access it as follows from the Ambari server node:

psql ambari ambari [default pw is bigdata]
select service_name, component_name, alert_state from alert_history where service_name='KAFKA' limit 5;

The results will look something like this:

service_name | component_name | alert_state
--------------+----------------+-------------
 KAFKA        | KAFKA_BROKER   | CRITICAL
 KAFKA        | KAFKA_BROKER   | OK
 KAFKA        | KAFKA_BROKER   | CRITICAL

You can get more details from that table and derive the information that you are looking for.

Re: How to view Alert History via Ambari ?

andrewg — Thu, 08 Oct 2015 22:10:50 GMT

Check out the /var/log/ambari-server/ambari-alerts.log file, it writes an alert stream to the file. There is also another way, potentially, I'll flag it with right team to comment on this thread next.

Re: How to view Alert History via Ambari ?

afernandez — Thu, 22 Oct 2015 04:29:25 GMT

Another method is to use the API.

http://server:8080/api/v1/clusters/<cluster_name>/alert_history

You can also filter for individual fields, or use query predicates like =, !=, in().

api/v1/clusters/c1/alert_history?fields/AlertHistory/

api/v1/clusters/c1/alerts?Alert/state=CRITICAL

Re: How to view Alert History via Ambari ?

andrewg — Thu, 22 Oct 2015 21:23:47 GMT

Based on @afernandez@hortonworks.com suggestion, reference docs for Alerts and Alerts History are available at https://github.c om/apache/ambari/blob/branch-2.1/ambari-server/docs/api/v1/alerts.md

E.g., to get critical Kafka service alerts (I am using a brilliant cli JSON processor, jq https://stedolan.github.io/jq/ 😞

curl -s -u admin:admin -H 'X-Requested-BY: ambari' \
http://localhost:8080/api/v1/clusters/Sandbox/services/KAFKA/alert_history?fields=* \

| jq '.items[] | select (.AlertHistory.state=="CRITICAL")'

This gives me an output like this (pretty-printed):

{
  "href": "http://localhost:8080/api/v1/clusters/Sandbox/services/KAFKA/alert_history/16",
  "AlertHistory": {
    "cluster_name": "Sandbox",
    "component_name": "KAFKA_BROKER",
    "definition_id": 1,
    "definition_name": "kafka_broker_process",
    "host_name": "sandbox.hortonworks.com",
    "id": 16,
    "instance": null,
    "label": "Kafka Broker Process",
    "service_name": "KAFKA",
    "state": "CRITICAL",
    "text": "Connection failed: [Errno 111] Connection refused to sandbox.hortonworks.com:6667",
    "timestamp": 1439987016669
  }
}
{
  "href": "http://localhost:8080/api/v1/clusters/Sandbox/services/KAFKA/alert_history/184",
  "AlertHistory": {
    "cluster_name": "Sandbox",
    "component_name": "KAFKA_BROKER",
    "definition_id": 1,
    "definition_name": "kafka_broker_process",
    "host_name": "sandbox.hortonworks.com",
    "id": 184,
    "instance": null,
    "label": "Kafka Broker Process",
    "service_name": "KAFKA",
    "state": "CRITICAL",
    "text": "Connection failed: [Errno 111] Connection refused to sandbox.hortonworks.com:6667",
    "timestamp": 1439989776335
  }
}
{
  "href": "http://localhost:8080/api/v1/clusters/Sandbox/services/KAFKA/alert_history/452",
  "AlertHistory": {
    "cluster_name": "Sandbox",
    "component_name": "KAFKA_BROKER",
    "definition_id": 1,
    "definition_name": "kafka_broker_process",
    "host_name": "sandbox.hortonworks.com",
    "id": 452,
    "instance": null,
    "label": "Kafka Broker Process",
    "service_name": "KAFKA",
    "state": "CRITICAL",
    "text": "Connection failed: [Errno 111] Connection refused to sandbox.hortonworks.com:6667",
    "timestamp": 1443742219799
  }
}

Re: How to view Alert History via Ambari ?

smdas — Fri, 23 Feb 2018 15:59:46 GMT

Thanks Brandon. I used the above approach to retrieve the history of alerts for a specific type.

Adding a small trick for anyone using the above approach. The table "Alert_History" has the Alert Timestamp column in BigInt. For readability, use the PSQL Cast functionality. In my example, I am using the SQL to identify when Ambari triggers the DataNode WebUI alerts across all DataNodes:

Select TO_CHAR(TO_TIMESTAMP(Alert_Timestamp / 1000), 'MM/DD/YYYY HH24:MI:SS') From Alert_History Where Alert_Label Like '%DataNode%UI%' And Alert_State ='CRITICAL' Order By 1 ASC;