Where is the state of the application master is persisted in YARN??

[#1]

The available state-store implementations are org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore , a ZooKeeper based state-store implementation and org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore , a Hadoop FileSystem based state-store implementation like HDFS. The default value is set to org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore. check zknode /rmstore under zookeeper

[#2]

@Rajkumar Singh

The state-store of RM is fine , will be in levelDB or zkRMstatestore or FSRMstatestore. My doubt is , about Appmaster.

As per my knowledge Appmaster will only notify RM about the status of the application, but not at the each task level. I have read that Appmaster will store its state in a persisted storage. What is this persisted storage exactly.

[#3]

To me, the AM state appears to be in memory. For example, the AM host and port are sent to the RM during a registration call.

When the AM is running, it is sending a heartbeat to the RM.

https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/yarn/applications/distributedshell/Appli...

When the job is done, the AM sends a FinishApplicationMasterRequest, which includes the final stage of the application master.

https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/yarn/api/protocolrecords/FinishApplicati...

Also, have you seen the following, which describes how to write an AM?

https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html#Writ...

Interesting question.

[#4]

Like @Rajkumar Singh mentioned above, the application state (like life cycle related events) get persistent in RM with RMStateStore. For ApplicationMaster (AM), there is no central place for AM to persistent its own internal state but it is flexible for AM to pickup places to store temporary results/progress so AM failed/restart won't have to lose all progress that last AM attempt gains. Take MapReduce for example, when configured properly, MR AM after restart will read finished map/reduce tasks from job history files on HDFS, so finished map/reduce tasks won't get re-executed after AM restart. Other application could have similar behavior in case it want to persist something.

[#5]

Junping Du

what is the word "places" in "flexible for AM to pickup places to store temporary results/progress" exactly refering to?/ RMstatestore?? or any other place ?? if any other place what is it and can it be configurable??