Community Articles

Find and share helpful community-sourced technical articles.
Celebrating as our community reaches 100,000 members! Thank you!
Labels (2)
Rising Star

As an extension to the article mentioned here we are using custom Ambari alerts to monitor the current state of the Journal Node edits health.

With the default monitoring that is present in the Ambari, we would not be alerted about the failure of edits that may happen in the one of the JN quorum. In typical HDFS HA env, there are three Journal node daemons that are deployed. If any one of the daemons fails to maintain the edits, then we are at risk of failovers and eventual cluster outage if another journal node hits similar issue as other journal node (Because, if quorum of edits are not maintained, then NN fails to be up). Hence, we need to have necessary alerting mechanism put in place for such failures. Journal Nodes may not get updated due to various reasons such as,

1. Disk getting full.
2. Corrupt Permissions.
3. Exhausted HDFS handlers in JN host, etc..

Attached are the artifacts, which contains,

1. alerts-test.json 
2. have preconfigured values,


Which defines the corresponding time ranges in seconds for alerts to be triggered. This would alert in Ambari, if the "edits_inprogress" file is not updated for above configured time interval.

Steps to configure the alert

1. Copy the to /var/lib/ambari-server/resources/host_scripts

2. Now restart the Ambari-Server.

3. Run the following command to list all the existing alerts:

curl -u admin:admin -i -H 'X-Requested-By:ambari' -X GET

4. Install the custom alert using Curl command as following:

curl -u admin:admin -i -H 'X-Requested-By:ambari' -X POST -d @alerts-test.json

Attachments :