Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

What is the difference between timeline server api and resource manager api calls for applications

avatar

1) hostname:8188/ws/v1/applicationhistor/apps

2) hostname:8088/ws/vi/cluster/apps

The first url is rest api call from timeline server and second url is from RM api. Both the url's are supposed to return list of ALL applications (succeeded/killed/failed) run on the cluster.

But there are a number of applications appearing in timeline server api call but NOT in RM api call.

Is this difference expected? What is the purpose/difference between these 2 approach of getting list of application run on cluster?

For example:

application_1477690913969_0092 and application_1477690913969_0093 appears in timeline svr api call but doesn't gets listed in RM api call.

Please see the attached screenshots.screen2.jpgscreen3.jpg

1 ACCEPTED SOLUTION

avatar
Master Guru

@Vishal Prakash Shah

I'm not 100% sure but I think this is expected as RM does not keep historical information of all the Applications. Main purpose of Yarn Application Time server is to maintain historical information(yarn.timeline-service.ttl-ms is the parameter for retention) about all the YARN jobs hence you see lot of results with Timeline API.

Default value of yarn.timeline-service.ttl-ms is 2678400000 ms i.e. 31 days

You can read more about timeline server here - https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/TimelineServer.html

Hope this information helps!

View solution in original post

5 REPLIES 5

avatar
Master Guru

@Vishal Prakash Shah

I'm not 100% sure but I think this is expected as RM does not keep historical information of all the Applications. Main purpose of Yarn Application Time server is to maintain historical information(yarn.timeline-service.ttl-ms is the parameter for retention) about all the YARN jobs hence you see lot of results with Timeline API.

Default value of yarn.timeline-service.ttl-ms is 2678400000 ms i.e. 31 days

You can read more about timeline server here - https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/TimelineServer.html

Hope this information helps!

avatar

In my setup, yarn.timeline-service.ttl-ms is configured with default value. Still I can see the both api's returning application details for more than 3 months. Any idea why??

avatar
Master Guru

@Vishal Prakash Shah - Can you please post value of your yarn.timeline-service.ttl-ms?

avatar

It is 2678400000 ttl.jpg

avatar

My question is-

if application_1477690913969_0092 is shown as an application being run on cluster, then why command yarn application -status application_1477690913969_0092 is giving error. The application was run on 30th Nov, 2016. It appears in timeline server api call but not in resource manager api call. There are results in rm api call which were run before 30th Nov.