Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
Super Mentor

This article talks about two things:

1. How to develop a simple python script which returns the result_code like 'CRITICAL', 'WARN', 'OK', 'UNKNOWN' based on certain conditions matching our requirement.

2. How to write an alert definition in json format so that we can install those alerts to the Ambari Server in order to get the alerts.

The scripts/files attached to this article are "test_alert_disk_space.py" and "alerts.json" that we are going to use as part of the following steps:

Step-1). Create a python script as attached "test_alert_disk_space.py"

that finds the list of mount points and the it returns the disk usage status of those mount points. The based on the percentages specified it returns the return code as

'CRITICAL', 'WARN', 'OK', 'UNKNOWN'

Step-2). Place the file "test_alert_disk_space.py" in the following Path on the ambari-server : "/var/lib/ambari-server/resources/host_scripts/"

Example:

cp -f test_alert_disk_space.py   /var/lib/ambari-server/resources/host_scripts

Step-3). Now restart the Ambari-Server.

Also we will need to restart the ambari-agents on each hosts so that it (agents) can pull the script from the ambari server. In this case when we restart ambari agents then the file "test_alert_disk_space.py" will be fetched by agents and will be stored inside the ambari-agent cache dir: "/var/lib/ambari-agent/cache/host_scripts" on agent hosts.

Step-4). Run the following command to list all the existing alerts:

curl -u admin:admin -i -H 'X-Requested-By:ambari' -X GET http://node1.example.com:8080/api/v1/clusters/ClusterDemo/alert_definitions

Here Ambari Server Host & port is : "node1.example.com:8080" and Cluster name is : "ClusterDemo"

Step-5). Install the custom alert using Curl command as following:

curl -u admin:admin -i -H 'X-Requested-By:ambari' -X POST -d @alerts.json  http://node1.example.com:8080/api/v1/clusters/ClusterDemo/alert_definitions

The output of the above command execution should have the response code as 201 (created). If the response code is 500 (Internal Server Error) or 404 ( Resource not fount) then please double check the command URL/json file.

Check the ambari console to find if the alerts are getting triggered fine or not. Alternatively run the command mentioned in Step-4 to verify of the custom alert is registered fine or not. If needed please do "ambari-server restart".

In the "test_alert_disk_space.py" script users can change the values of "PERCENT_USED_WARNING_KEY" and "PERCENT_USED_CRITICAL_KEY" as per their requirement.

# script parameter keys
MIN_FREE_SPACE_KEY = "minimum.free.space"
PERCENT_USED_WARNING_KEY = "percent.used.space.warning.threshold"
PERCENT_USED_CRITICAL_KEY = "percent.free.space.critical.threshold"

PERCENT_USED_WARNING_KEY = 60
PERCENT_USED_CRITICAL_KEY = 80

Any changes if we make in this file then the ambari-server restart is needed so that ambari-server can push those changes to the agent hosts.

We should be able to see the following kind of alerts based on the the should we have set:

4799-ambari-alert-1.png

List of every mount point and it's disk usage percentage.

4800-ambari-alert-2.png

[OPTIONAL]

Manually Running Alert

If we want to manually run the alert then do the following (Notice the "?run_now=true" part in the url)

curl -u admin:admin -i -H 'X-Requested-By:ambari' -X PUT  http://node1.example.com:8080/api/v1/clusters/ClusterDemo/alert_definitions/151?run_now=true

Notice here the "151 is the "alert id" which we can get from the ambari console alert definitions page. or using the curl GET command from Step-4.

Deleting the Alerts
11,411 Views
Comments
Super Mentor

Remember that users might do mistake in changing the threshold of alert in "PERCENT_USED_WARNING_DEFAULT" (default value), Instead of the correct variable "PERCENT_USED_WARNING_KEY"

New Contributor

Hi Jay SenSharma.

This is a great tutorial. You provided a complete tutorial. I followed up the steps and it seems alerts are being generated for disk usage.

I have a question about the json file format. If I would create my own alerts I need to know what are the required fields and what are the expected values? Do you have a documentation?

Regards,

Alex Feldman

New Contributor

You can see examples of the various alert definitions by getting them using API. The JSON returned can be used as a template to create your own definition. https://docs.daplab.ch/ambari_cheat_sheet/ eg.

curl -v -X GET -u ${ambari_credentials} -H 'X-Requested-By:ambari' https://admin.daplab.ch/api/v1/clusters/DAPLAB02/alert_definitions?AlertDefinition/service_name=HIVE

The 5 alert types are here.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.1/bk_Ambari_Users_Guide/content/_alert_defini...

New Contributor

Hi Jay SenSharma.

After some research these are my latest comments. Since this is the community site my comments are intended to be helpful to other users of this portal.

While the Short Description promised to cover two things, none was actually covered.

1. How to develop a simple python script which returns the result code like 'CRITICAL', 'WARN', 'OK', 'UNKNOWN' based on certain conditions matching our requirement.

This point was never covered since no script was developed. The script was provided to complete this exercise. It was not mentioned that the script should be written using Python. The statement “a simple python script which returns the … code” is misleading, since script does not return any strings listed. The method exec does…when it is being called. It seems that the script must contain the exec method that is being called by ambary framework.

In addition, the method exec does not return just a token string… It returns a json with a listed token being one of the values…

2. How to write an alert definition in json format so that we can install those alerts to the Ambari Server in order to get the alerts.

This point is not covered as well. See my prior posting. Considering that we only have 4 API calls to play with: GET, POST, PUT, DELETE to list required fields used in json and their expected values would be appropriate.

Good Samaritan Andrew Sears provided the link to Ambari Cheat Sheet site… the example lists a shell script to be used and python script with the identical name in the json to be used in API call… for more confusion I presume.

On the other hand it lists json that is relatively easy to “comprehend” by initiated.

Still what is the meaning of the fields "ignore_host"? "component_name"? "service_name"? "scope"?

In conclusion: for the article to be helpful it needs more actual details.

Regards,

Alex Feldman

New Contributor

@Jay SenSharmaHi, I almost developed my script and then found this. Why can't we simply execute df -h from python and parse the result instead of doing all the calculations? Any problem with this approach?

New Contributor

Thanks for the awesome article.

When I follow your example, everything works fine. However, it I change a script an restart ambari-server, the changes will not be propagated to the amber-agents, I have to restart the agent manually to trigger a sync.

Is there a sync interval setting for the agents?

Super Mentor

@Simon Heeg

Actually we will need to restart the ambari agents on all the hosts, so that during the restart of agents they will pull the scripts(or changes in the script) from ambari server. I have fixed the Step3) as following:

Step-3). Now restart the Ambari-Server.

Also we will need to restart the ambari-agents on each hosts so that it (agents) can pull the script from the ambari server. In this case when we restart ambari agents then the file "test_alert_disk_space.py" will be fetched by agents and will be stored inside the ambari-agent cache dir: "/var/lib/ambari-agent/cache/host_scripts" on agent hosts.

.

New Contributor

Thanks Jay for your awesome support,

it would be very helpful if you could provide information on how to access ambari configuration in service checks. When I let ambari print the configurations and parameters variables of the execute function, then configurations is empty and parameters contains my parameters as well as "kerberos.kinit.timer": 14400000.

EDIT: I found the answer by digging through the Ambari source code, if anyone else is struggling, see: Ambari @GitHub

Is that only me can't extract the downloaded custom-alerts.zip?

Super Mentor

@Hajime

I just tried downloading the "4811-custom-alerts.zip" and i can extract it properly. How are you downloading it.

$ md5 4811-custom-alerts.zip 
MD5 (4811-custom-alerts.zip) = a33f105860d07e05149b68960a0ea0c9

.

figured out: jar xvf 4811-custom-alerts.zip

New Contributor

Can we use test_alert_disk_space.sh file at place of test_alert_disk_space.py ?

Contributor

Hi All!

After done steps describe above have strange errors : [Errno 2] No such file or directory: 'net_prio'

Please healp custom-host-mount-point-usage.png..

Ambari Version 2.5.0.3 (HDP-2.6.3.0)

New Contributor

The script presented here doesn't work on CentOS 7 + Python 7 + Ambari 2.6.2.2.

See https://community.hortonworks.com/questions/160781/custom-ambari-alerts-error-custom-ambari-alerts.h... for additional discussion.

Not applicable

So, I have added web type alerts to check some URLs. I have uploaded them into Ambari, assigned them to a group and manually run them with a PUT like http://localhost:8080/api/v1/clusters/Magellan/alert_definitions/313?run_now=true. They respond correctly.

But for the life of me I cannot see how to automate them (aside from a cron job that keeps applying the curl command to manually execute each one.)

I've tried intervals of 1 & 2 without any luck. Is there some sort of heartbeat or trigger that I am supposed to assign to them?

Thanks for any ideas.

New Contributor

Referring to the PUT command used for triggering the alert manually, Is it possible to pass some parameters/custom information to the script which gets trigged by the alert ? If yes, is it via headers OR body of the PUT request ?

Thanks.

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 12:03 PM
Updated by:
 
Contributors
Top Kudoed Authors