question Re: Yarn App Timeline Server down quietly in Archives of Support Questions (Read Only)

Yarn App Timeline Server down quietly

richard_xu — Tue, 22 Mar 2016 00:31:48 GMT

No error/exception found in log, only a few lines of warning:

2016-03-21 06:49:49,236 INFO timeline.LogInfo (LogInfo.java:parseForStore(76)) - Parsed 0 entities from hdfs://hostname/ats/active/application_1458130432014_0671/appattempt_1458130432014_0671_000001/summarylog-appattempt_1458130432014_0671_000001 in 11 msec

2016-03-21 06:49:49,246 WARN timeline.EntityGroupFSTimelineStore (LogInfo.java:doParse(164)) - Error putting entity: dag_1458130432014_0695_1 (TEZ_DAG_ID): 6

2016-03-21 06:49:49,246 WARN timeline.EntityGroupFSTimelineStore (LogInfo.java:doParse(164)) - Error putting entity: dag_1458130432014_0694_1 (TEZ_DAG_ID): 6

2016-03-21 06:49:49,247 INFO timeline.LogInfo (LogInfo.java:parseForStore(76)) - Parsed 6 entities from hdfs://hostname/ats/active/application_1458130432014_0695/appattempt_1458130432014_0695_000001/summarylog-appattempt_1458130432014_0695_000001 in 19 msec

2016-03-21 06:49:49,247 INFO timeline.LogInfo (LogInfo.java:parseForStore(76)) - Parsed 6 entities from hdfs://hostname/ats/active/application_1458130432014_0694/appattempt_1458130432014_0694_000001/summarylog-appattempt_1458130432014_0694_000001 in 19 msec

It is HDP 2.3.4.

Re: Yarn App Timeline Server down quietly

SK1 — Tue, 22 Mar 2016 20:44:38 GMT

Can you restart all mapreduce and Yarn services ?

Re: Yarn App Timeline Server down quietly

richard_xu — Thu, 24 Mar 2016 21:31:23 GMT

We are able to restart Yarn App Timeline server. Further investigation indicates it leaks memory: yarn 32397 8.4 43.6 8432880 6680964 ? Sl Mar23 101:19

/usr/jdk64/jdk1.8.0_60/bin/java -Dproc_timelineserver -Xmx1024m -Dhdp.version=2.3.4.0-3485

Configured with 1GB max but now using 6GB, thought?

Re: Yarn App Timeline Server down quietly

agillan — Wed, 30 Mar 2016 23:07:53 GMT

Hi @Richard Xu, there is a known issue in HDP 2.3.4.0 where the application timeline server leaks threads.

See:

https://issues.apache.org/jira/browse/TEZ-3017

https://issues.apache.org/jira/browse/HIVE-12766

The thread leak is fixed in HDP 2.3.4.7, so the recommendation is to upgrade to that version or higher. You can also disable ATS 1.5 and downgrade to ATS 1.0, but the upgrade is recommended.