Why am I seeing this odd behaviour? YARN / App Timeline Web UI: Connection Failed

[#1]

Hi All,

Recently I setup email notifications in Ambari (2.2.1.0) to receive notifications when alert states change. I've found that throughout each day the YARN alert App Timeline Web UI: Connection Failed switches from CRITICAL (connection timed out on port 8188) to OK. This happens 12 / 14 times a day. I'm unsure why this is? I have a basic 2 node cluster, the App Timeline and History Server components are on node 1, and the Resource Manager is on node 1 if that helps.

Any thoughts on why this might happen - it doesn't seem to effect performance.

Mike

[#2 · Accepted solution]

Hi @mike harding

This seems to be known behaviour in Ambari 2.2.1.0. There is also internal bug raised in hortonworks jira for the same - BUG-47724.

This jira is resolved and the resolution seems to be fixed in Ambari 2.2.2.0

[#3]

Meanwhile can you check if you are able to get JMX value of the alert. Also try to get the alert defination using api -

Make a GET request to get the alert definition for the alert you want to change

• GET api/v1/clusters/<cluster>/alert_definitions
• GET api/v1/clusters/<cluster>/alert_definitions/<id> from above call

[#4]

@mike harding

You can try 2 options -

1. You can also try modifying the "Check Interval" for this service by logging to - Ambari UI->Alerts-"App Timeline Web UI"->Edit -> modify "Check Interval" to higher value.

2.You can try modifying "connection_timeout" value for the timeline webui. Please follow steps below

a.Check alert defination using below command -

http://<ambari_fqdn>:8080/api/v1/clusters/<cluster-name>/alert_definitions/

b.Get the respective alert definition you want to modify from above command- here for example we will modify "App Timeline webui" alert value for "connection_timeout" changing value from 5.0 to 50.0

c. Copy the json into file name test.json

d. Modify the json value for "connection_timeout" : 5.0" to "connection_timeout" : 50.0" and save the file.

e. PUT it using below command -

curl -H 'X-Requested-By:ambari' -u $ambari_username:$ambari_password -X PUT --data @test.json http://<ambari_fqdn>:8080/api/v1/clusters/<cluster-name>/alert_definitions/<alert_no>/

[#5]

If you have Kerberos enabled, you might be hitting: https://issues.apache.org/jira/browse/AMBARI-14847

You can upgrade to Ambari 2.2.2 and that should resolve the issue.