What is the difference between timeline server api and resource manager api calls for applications

[#1]

1) hostname:8188/ws/v1/applicationhistor/apps

2) hostname:8088/ws/vi/cluster/apps

The first url is rest api call from timeline server and second url is from RM api. Both the url's are supposed to return list of ALL applications (succeeded/killed/failed) run on the cluster.

But there are a number of applications appearing in timeline server api call but NOT in RM api call.

Is this difference expected? What is the purpose/difference between these 2 approach of getting list of application run on cluster?

For example:

application_1477690913969_0092 and application_1477690913969_0093 appears in timeline svr api call but doesn't gets listed in RM api call.

Please see the attached screenshots.screen2.jpgscreen3.jpg

[#2 · Accepted solution]

@Vishal Prakash Shah

I'm not 100% sure but I think this is expected as RM does not keep historical information of all the Applications. Main purpose of Yarn Application Time server is to maintain historical information(yarn.timeline-service.ttl-ms is the parameter for retention) about all the YARN jobs hence you see lot of results with Timeline API.

Default value of yarn.timeline-service.ttl-ms is 2678400000 ms i.e. 31 days

You can read more about timeline server here - https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/TimelineServer.html

Hope this information helps!

[#3]

In my setup, yarn.timeline-service.ttl-ms is configured with default value. Still I can see the both api's returning application details for more than 3 months. Any idea why??

[#4]

@Vishal Prakash Shah - Can you please post value of your yarn.timeline-service.ttl-ms?

[#5]

It is 2678400000 ttl.jpg

[#6]

My question is-

if application_1477690913969_0092 is shown as an application being run on cluster, then why command yarn application -status application_1477690913969_0092 is giving error. The application was run on 30th Nov, 2016. It appears in timeline server api call but not in resource manager api call. There are results in rm api call which were run before 30th Nov.