Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to view Alert History via Ambari ?

Solved Go to solution
Highlighted

How to view Alert History via Ambari ?

I want to view how many times Kafka died in the last 10 days . Is there any ways to view it from Ambari ?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to view Alert History via Ambari ?

Based on @afernandez@hortonworks.com suggestion, reference docs for Alerts and Alerts History are available at https://github.com/apache/ambari/blob/branch-2.1/ambari-server/docs/api/v1/alerts.md

E.g., to get critical Kafka service alerts (I am using a brilliant cli JSON processor, jq https://stedolan.github.io/jq/ ):

curl -s -u admin:admin -H 'X-Requested-BY: ambari' \
http://localhost:8080/api/v1/clusters/Sandbox/services/KAFKA/alert_history?fields=* \

| jq '.items[] | select (.AlertHistory.state=="CRITICAL")'

This gives me an output like this (pretty-printed):

{
  "href": "http://localhost:8080/api/v1/clusters/Sandbox/services/KAFKA/alert_history/16",
  "AlertHistory": {
    "cluster_name": "Sandbox",
    "component_name": "KAFKA_BROKER",
    "definition_id": 1,
    "definition_name": "kafka_broker_process",
    "host_name": "sandbox.hortonworks.com",
    "id": 16,
    "instance": null,
    "label": "Kafka Broker Process",
    "service_name": "KAFKA",
    "state": "CRITICAL",
    "text": "Connection failed: [Errno 111] Connection refused to sandbox.hortonworks.com:6667",
    "timestamp": 1439987016669
  }
}
{
  "href": "http://localhost:8080/api/v1/clusters/Sandbox/services/KAFKA/alert_history/184",
  "AlertHistory": {
    "cluster_name": "Sandbox",
    "component_name": "KAFKA_BROKER",
    "definition_id": 1,
    "definition_name": "kafka_broker_process",
    "host_name": "sandbox.hortonworks.com",
    "id": 184,
    "instance": null,
    "label": "Kafka Broker Process",
    "service_name": "KAFKA",
    "state": "CRITICAL",
    "text": "Connection failed: [Errno 111] Connection refused to sandbox.hortonworks.com:6667",
    "timestamp": 1439989776335
  }
}
{
  "href": "http://localhost:8080/api/v1/clusters/Sandbox/services/KAFKA/alert_history/452",
  "AlertHistory": {
    "cluster_name": "Sandbox",
    "component_name": "KAFKA_BROKER",
    "definition_id": 1,
    "definition_name": "kafka_broker_process",
    "host_name": "sandbox.hortonworks.com",
    "id": 452,
    "instance": null,
    "label": "Kafka Broker Process",
    "service_name": "KAFKA",
    "state": "CRITICAL",
    "text": "Connection failed: [Errno 111] Connection refused to sandbox.hortonworks.com:6667",
    "timestamp": 1443742219799
  }
}
5 REPLIES 5

Re: How to view Alert History via Ambari ?

I am not aware of a historical view into this information via the Ambari UI but you can certainly grab this information via the backing Ambari database. Using the default installation you can access it as follows from the Ambari server node:

  • psql ambari ambari [default pw is bigdata]
  • select service_name, component_name, alert_state from alert_history where service_name='KAFKA' limit 5;

The results will look something like this:

service_name | component_name | alert_state
--------------+----------------+-------------
 KAFKA        | KAFKA_BROKER   | CRITICAL
 KAFKA        | KAFKA_BROKER   | OK
 KAFKA        | KAFKA_BROKER   | CRITICAL

You can get more details from that table and derive the information that you are looking for.

Re: How to view Alert History via Ambari ?

Cloudera Employee

Thanks Brandon. I used the above approach to retrieve the history of alerts for a specific type.

Adding a small trick for anyone using the above approach. The table "Alert_History" has the Alert Timestamp column in BigInt. For readability, use the PSQL Cast functionality. In my example, I am using the SQL to identify when Ambari triggers the DataNode WebUI alerts across all DataNodes:

Select TO_CHAR(TO_TIMESTAMP(Alert_Timestamp / 1000), 'MM/DD/YYYY HH24:MI:SS') From Alert_History Where Alert_Label Like '%DataNode%UI%' And Alert_State ='CRITICAL' Order By 1 ASC;

Re: How to view Alert History via Ambari ?

Check out the /var/log/ambari-server/ambari-alerts.log file, it writes an alert stream to the file. There is also another way, potentially, I'll flag it with right team to comment on this thread next.

Re: How to view Alert History via Ambari ?

Another method is to use the API.

http://server:8080/api/v1/clusters/<cluster_name>/alert_history

You can also filter for individual fields, or use query predicates like =, !=, in().

api/v1/clusters/c1/alert_history?fields/AlertHistory/

api/v1/clusters/c1/alerts?Alert/state=CRITICAL

Re: How to view Alert History via Ambari ?

Based on @afernandez@hortonworks.com suggestion, reference docs for Alerts and Alerts History are available at https://github.com/apache/ambari/blob/branch-2.1/ambari-server/docs/api/v1/alerts.md

E.g., to get critical Kafka service alerts (I am using a brilliant cli JSON processor, jq https://stedolan.github.io/jq/ ):

curl -s -u admin:admin -H 'X-Requested-BY: ambari' \
http://localhost:8080/api/v1/clusters/Sandbox/services/KAFKA/alert_history?fields=* \

| jq '.items[] | select (.AlertHistory.state=="CRITICAL")'

This gives me an output like this (pretty-printed):

{
  "href": "http://localhost:8080/api/v1/clusters/Sandbox/services/KAFKA/alert_history/16",
  "AlertHistory": {
    "cluster_name": "Sandbox",
    "component_name": "KAFKA_BROKER",
    "definition_id": 1,
    "definition_name": "kafka_broker_process",
    "host_name": "sandbox.hortonworks.com",
    "id": 16,
    "instance": null,
    "label": "Kafka Broker Process",
    "service_name": "KAFKA",
    "state": "CRITICAL",
    "text": "Connection failed: [Errno 111] Connection refused to sandbox.hortonworks.com:6667",
    "timestamp": 1439987016669
  }
}
{
  "href": "http://localhost:8080/api/v1/clusters/Sandbox/services/KAFKA/alert_history/184",
  "AlertHistory": {
    "cluster_name": "Sandbox",
    "component_name": "KAFKA_BROKER",
    "definition_id": 1,
    "definition_name": "kafka_broker_process",
    "host_name": "sandbox.hortonworks.com",
    "id": 184,
    "instance": null,
    "label": "Kafka Broker Process",
    "service_name": "KAFKA",
    "state": "CRITICAL",
    "text": "Connection failed: [Errno 111] Connection refused to sandbox.hortonworks.com:6667",
    "timestamp": 1439989776335
  }
}
{
  "href": "http://localhost:8080/api/v1/clusters/Sandbox/services/KAFKA/alert_history/452",
  "AlertHistory": {
    "cluster_name": "Sandbox",
    "component_name": "KAFKA_BROKER",
    "definition_id": 1,
    "definition_name": "kafka_broker_process",
    "host_name": "sandbox.hortonworks.com",
    "id": 452,
    "instance": null,
    "label": "Kafka Broker Process",
    "service_name": "KAFKA",
    "state": "CRITICAL",
    "text": "Connection failed: [Errno 111] Connection refused to sandbox.hortonworks.com:6667",
    "timestamp": 1443742219799
  }
}
Don't have an account?
Coming from Hortonworks? Activate your account here