Setting up Ambari Alert for HiveServer2 Established Connections

[#1]

Current version of Ambari doesn't monitor Number of Hiveserver2 connections. We often see Hiveserver2 slowness due to heavy load as part increase in number connections to Hiveserver2. Setting up a alert for Hiveserver2 established connections will help us to take required actions like adding additional Hiveserver2 services, proper load balancing or scheduling the jobs.

NOTE : Please go through this article https://github.com/apache/ambari/blob/2.6.2-maint/ambari-server/docs/api/v1/alert-definitions.md to understand Ambari Alert Definition

Please find the python script and .json file used below in the attachments.

1. alert_hiveserver_num_connection.py - Is the python script that finds the current established connection for each Hiveserver2 and based on number of connection it returns 'CRITICAL', 'WARN', 'OK’
2. alerths.json - Is the Ambari Alert definition

Below are the steps to setup the Ambari Alert on Hiveserver2 Established connections.

Step 1 - Place the file “alert_hiveserver_num_connection.py" in the following path on the ambari-server : "/var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/alerts/ "

[root@vb-atlas-ambari tmp]# cp alert_hiveserver_num_connection.py /var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/alerts/ 

Step 2 - Restart Ambari Server, to force Ambari agents to pull alert_hiveserver_num_connection.py python script to every host.

ambari-server restart

Once Ambari Server is restarted , we can verify if alert_hiveserver_num_connection.py is available in " /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/ " location on Hiveserver2 host.

Note : Some time it takes longer for Ambari agent to pull the script from Ambari server.

[root@vb-atlas-node1 ~]# ll /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/

total 116

-rw-r--r--. 1 root root 9740 Jun 27 17:01 alert_hive_interactive_thrift_port.py

-rw-r--r--. 1 root root 7893 Jun 27 17:01 alert_hive_interactive_thrift_port.pyo

-rw-r--r--. 1 root root 9988 Jun 27 17:01 alert_hive_metastore.py

-rw-r--r--. 1 root root 9069 Jun 27 17:01 alert_hive_metastore.pyo

-rw-r--r--. 1 root root 1888 Jun 27 17:01 alert_hiveserver_num_connection.py

-rw-r--r--. 1 root root 11459 Jun 27 17:01 alert_hive_thrift_port.py

-rw-r--r--. 1 root root 9362 Jun 27 17:01 alert_hive_thrift_port.pyo

-rw-r--r--. 1 root root 11946 Jun 27 17:01 alert_llap_app_status.py

-rw-r--r--. 1 root root 9339 Jun 27 17:01 alert_llap_app_status.pyo

-rw-r--r--. 1 root root 8886 Jun 27 17:01 alert_webhcat_server.py

-rw-r--r--. 1 root root 6563 Jun 27 17:01 alert_webhcat_server.pyo 

Step 3 - Post the Alert Definition (alerths.json) to Ambari using curl

curl -u <Ambari_admin_username>:<Amabri_admin_password> -i -h 'X-Requested-By:ambari' -X POST -d @alerths.json http://<AMBARI_HOST>:<AMBARI_PORT>/api/v1/clusters/<CLUSTER_NAME>/alert_definitions

Example :

[root@vb-atlas-ambari tmp]# curl -u admin:admin -i -H 'X-Requested-By:ambari' -X POST -d @alerths.json http://172.26.108.142:8080/api/v1/clusters/vinod/alert_definitions

HTTP/1.1 201 Created

X-Frame-Options: DENY

X-XSS-Protection: 1; mode=block

X-Content-Type-Options: nosniff

Cache-Control: no-store

Pragma: no-cache

Set-Cookie: AMBARISESSIONID=10f33laf224yy1834ygq9cekbo;Path=/;HttpOnly

Expires: Thu, 01 Jan 1970 00:00:00 GMT

User: admin

Content-Type: text/plain

Content-Length: 0 

We should be able to see the Alert in Ambari -> Alerts ( HiveServer2 Established Connections)

Alternative we can also see the "HiveServer2 Established Connections" listed in Alert definitions “ http://<AMBARI_HOST>:<AMBARI_PORT>/api/v1/clusters/<CLUSTER_NAME>/alert_definitions “

Step 4 - As per Alert Definition (alerths.json) CRITICAL alert is set to 50 and WARNING is set to 30 connections by default. You can update the values directly from Ambari by editing the values.