Yarn App Timeline Server down quietly

[#1]

No error/exception found in log, only a few lines of warning:

2016-03-21 06:49:49,236 INFO timeline.LogInfo (LogInfo.java:parseForStore(76)) - Parsed 0 entities from hdfs://hostname/ats/active/application_1458130432014_0671/appattempt_1458130432014_0671_000001/summarylog-appattempt_1458130432014_0671_000001 in 11 msec

2016-03-21 06:49:49,246 WARN timeline.EntityGroupFSTimelineStore (LogInfo.java:doParse(164)) - Error putting entity: dag_1458130432014_0695_1 (TEZ_DAG_ID): 6

2016-03-21 06:49:49,246 WARN timeline.EntityGroupFSTimelineStore (LogInfo.java:doParse(164)) - Error putting entity: dag_1458130432014_0695_1 (TEZ_DAG_ID): 6

2016-03-21 06:49:49,246 WARN timeline.EntityGroupFSTimelineStore (LogInfo.java:doParse(164)) - Error putting entity: dag_1458130432014_0694_1 (TEZ_DAG_ID): 6

2016-03-21 06:49:49,246 WARN timeline.EntityGroupFSTimelineStore (LogInfo.java:doParse(164)) - Error putting entity: dag_1458130432014_0694_1 (TEZ_DAG_ID): 6

2016-03-21 06:49:49,247 INFO timeline.LogInfo (LogInfo.java:parseForStore(76)) - Parsed 6 entities from hdfs://hostname/ats/active/application_1458130432014_0695/appattempt_1458130432014_0695_000001/summarylog-appattempt_1458130432014_0695_000001 in 19 msec

2016-03-21 06:49:49,247 INFO timeline.LogInfo (LogInfo.java:parseForStore(76)) - Parsed 6 entities from hdfs://hostname/ats/active/application_1458130432014_0694/appattempt_1458130432014_0694_000001/summarylog-appattempt_1458130432014_0694_000001 in 19 msec

It is HDP 2.3.4.

[#2]

Can you restart all mapreduce and Yarn services ?

[#3]

We are able to restart Yarn App Timeline server. Further investigation indicates it leaks memory: yarn 32397 8.4 43.6 8432880 6680964 ? Sl Mar23 101:19

/usr/jdk64/jdk1.8.0_60/bin/java -Dproc_timelineserver -Xmx1024m -Dhdp.version=2.3.4.0-3485

Configured with 1GB max but now using 6GB, thought?

[#4 · Accepted solution]

Hi @Richard Xu, there is a known issue in HDP 2.3.4.0 where the application timeline server leaks threads.

See:

https://issues.apache.org/jira/browse/TEZ-3017

https://issues.apache.org/jira/browse/HIVE-12766

The thread leak is fixed in HDP 2.3.4.7, so the recommendation is to upgrade to that version or higher. You can also disable ATS 1.5 and downgrade to ATS 1.0, but the upgrade is recommended.