Support Questions
Find answers, ask questions, and share your expertise

Yarn App Timeline Server down quietly

Explorer

No error/exception found in log, only a few lines of warning:

2016-03-21 06:49:49,236 INFO timeline.LogInfo (LogInfo.java:parseForStore(76)) - Parsed 0 entities from hdfs://hostname/ats/active/application_1458130432014_0671/appattempt_1458130432014_0671_000001/summarylog-appattempt_1458130432014_0671_000001 in 11 msec

2016-03-21 06:49:49,246 WARN timeline.EntityGroupFSTimelineStore (LogInfo.java:doParse(164)) - Error putting entity: dag_1458130432014_0695_1 (TEZ_DAG_ID): 6

2016-03-21 06:49:49,246 WARN timeline.EntityGroupFSTimelineStore (LogInfo.java:doParse(164)) - Error putting entity: dag_1458130432014_0695_1 (TEZ_DAG_ID): 6

2016-03-21 06:49:49,246 WARN timeline.EntityGroupFSTimelineStore (LogInfo.java:doParse(164)) - Error putting entity: dag_1458130432014_0694_1 (TEZ_DAG_ID): 6

2016-03-21 06:49:49,246 WARN timeline.EntityGroupFSTimelineStore (LogInfo.java:doParse(164)) - Error putting entity: dag_1458130432014_0694_1 (TEZ_DAG_ID): 6

2016-03-21 06:49:49,247 INFO timeline.LogInfo (LogInfo.java:parseForStore(76)) - Parsed 6 entities from hdfs://hostname/ats/active/application_1458130432014_0695/appattempt_1458130432014_0695_000001/summarylog-appattempt_1458130432014_0695_000001 in 19 msec

2016-03-21 06:49:49,247 INFO timeline.LogInfo (LogInfo.java:parseForStore(76)) - Parsed 6 entities from hdfs://hostname/ats/active/application_1458130432014_0694/appattempt_1458130432014_0694_000001/summarylog-appattempt_1458130432014_0694_000001 in 19 msec

It is HDP 2.3.4.

1 ACCEPTED SOLUTION

Hi @Richard Xu, there is a known issue in HDP 2.3.4.0 where the application timeline server leaks threads.

See:

https://issues.apache.org/jira/browse/TEZ-3017

https://issues.apache.org/jira/browse/HIVE-12766

The thread leak is fixed in HDP 2.3.4.7, so the recommendation is to upgrade to that version or higher. You can also disable ATS 1.5 and downgrade to ATS 1.0, but the upgrade is recommended.

View solution in original post

3 REPLIES 3

Guru

Can you restart all mapreduce and Yarn services ?

Explorer

We are able to restart Yarn App Timeline server. Further investigation indicates it leaks memory: yarn 32397 8.4 43.6 8432880 6680964 ? Sl Mar23 101:19

/usr/jdk64/jdk1.8.0_60/bin/java -Dproc_timelineserver -Xmx1024m -Dhdp.version=2.3.4.0-3485

Configured with 1GB max but now using 6GB, thought?

Hi @Richard Xu, there is a known issue in HDP 2.3.4.0 where the application timeline server leaks threads.

See:

https://issues.apache.org/jira/browse/TEZ-3017

https://issues.apache.org/jira/browse/HIVE-12766

The thread leak is fixed in HDP 2.3.4.7, so the recommendation is to upgrade to that version or higher. You can also disable ATS 1.5 and downgrade to ATS 1.0, but the upgrade is recommended.

; ;