How to view Alert History via Ambari ?

[#1]

I want to view how many times Kafka died in the last 10 days . Is there any ways to view it from Ambari ?

[#2]

I am not aware of a historical view into this information via the Ambari UI but you can certainly grab this information via the backing Ambari database. Using the default installation you can access it as follows from the Ambari server node:

• psql ambari ambari [default pw is bigdata]
• select service_name, component_name, alert_state from alert_history where service_name='KAFKA' limit 5;

The results will look something like this:

service_name | component_name | alert_state
--------------+----------------+-------------
 KAFKA        | KAFKA_BROKER   | CRITICAL
 KAFKA        | KAFKA_BROKER   | OK
 KAFKA        | KAFKA_BROKER   | CRITICAL

You can get more details from that table and derive the information that you are looking for.

[#3]

Thanks Brandon. I used the above approach to retrieve the history of alerts for a specific type.

Adding a small trick for anyone using the above approach. The table "Alert_History" has the Alert Timestamp column in BigInt. For readability, use the PSQL Cast functionality. In my example, I am using the SQL to identify when Ambari triggers the DataNode WebUI alerts across all DataNodes:

Select TO_CHAR(TO_TIMESTAMP(Alert_Timestamp / 1000), 'MM/DD/YYYY HH24:MI:SS') From Alert_History Where Alert_Label Like '%DataNode%UI%' And Alert_State ='CRITICAL' Order By 1 ASC;

[#4]

Check out the /var/log/ambari-server/ambari-alerts.log file, it writes an alert stream to the file. There is also another way, potentially, I'll flag it with right team to comment on this thread next.

[#5]

Another method is to use the API.

http://server:8080/api/v1/clusters/<cluster_name>/alert_history

You can also filter for individual fields, or use query predicates like =, !=, in().

api/v1/clusters/c1/alert_history?fields/AlertHistory/

api/v1/clusters/c1/alerts?Alert/state=CRITICAL

[#6 · Accepted solution]

Based on @afernandez@hortonworks.com suggestion, reference docs for Alerts and Alerts History are available at https://github.com/apache/ambari/blob/branch-2.1/ambari-server/docs/api/v1/alerts.md

E.g., to get critical Kafka service alerts (I am using a brilliant cli JSON processor, jq https://stedolan.github.io/jq/ 😞

curl -s -u admin:admin -H 'X-Requested-BY: ambari' \
http://localhost:8080/api/v1/clusters/Sandbox/services/KAFKA/alert_history?fields=* \

| jq '.items[] | select (.AlertHistory.state=="CRITICAL")'

This gives me an output like this (pretty-printed):

{
  "href": "http://localhost:8080/api/v1/clusters/Sandbox/services/KAFKA/alert_history/16",
  "AlertHistory": {
    "cluster_name": "Sandbox",
    "component_name": "KAFKA_BROKER",
    "definition_id": 1,
    "definition_name": "kafka_broker_process",
    "host_name": "sandbox.hortonworks.com",
    "id": 16,
    "instance": null,
    "label": "Kafka Broker Process",
    "service_name": "KAFKA",
    "state": "CRITICAL",
    "text": "Connection failed: [Errno 111] Connection refused to sandbox.hortonworks.com:6667",
    "timestamp": 1439987016669
  }
}
{
  "href": "http://localhost:8080/api/v1/clusters/Sandbox/services/KAFKA/alert_history/184",
  "AlertHistory": {
    "cluster_name": "Sandbox",
    "component_name": "KAFKA_BROKER",
    "definition_id": 1,
    "definition_name": "kafka_broker_process",
    "host_name": "sandbox.hortonworks.com",
    "id": 184,
    "instance": null,
    "label": "Kafka Broker Process",
    "service_name": "KAFKA",
    "state": "CRITICAL",
    "text": "Connection failed: [Errno 111] Connection refused to sandbox.hortonworks.com:6667",
    "timestamp": 1439989776335
  }
}
{
  "href": "http://localhost:8080/api/v1/clusters/Sandbox/services/KAFKA/alert_history/452",
  "AlertHistory": {
    "cluster_name": "Sandbox",
    "component_name": "KAFKA_BROKER",
    "definition_id": 1,
    "definition_name": "kafka_broker_process",
    "host_name": "sandbox.hortonworks.com",
    "id": 452,
    "instance": null,
    "label": "Kafka Broker Process",
    "service_name": "KAFKA",
    "state": "CRITICAL",
    "text": "Connection failed: [Errno 111] Connection refused to sandbox.hortonworks.com:6667",
    "timestamp": 1443742219799
  }
}