can applicationMaster directly communicate with containers(once they got allocated) which resides in different nodes? If yes , how? Thank you.

[#1]

Hi @Gunjan Dhawas,

No, ApplicationMaster doesnot directly communicate with the containers. Its the nodemanagers which communicate directly with the containers. NodeManager's core part is containerManagement.

ApplicationMaster, framework specific entity is responsible for negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the containers and their resource consumption

More information on Nodemanager and AM - https://hortonworks.com/blog/apache-hadoop-yarn-nodemanager/

https://hortonworks.com/blog/apache-hadoop-yarn-concepts-and-applications/

[#2]

@ssathish Thank you! I'm having few more doubts,

1. As mentioned in this link https://hortonworks.com/blog/apache-hadoop-yarn-concepts-and-applications/ it says "The application code executing within the container then provides necessary information (progress, status etc.) to its ApplicationMaster via an application-specific protocol.", so how the application master monitor the status of containers which are running on different node than applicatioMaster.

2. Also as you mentioned its nodemanager which will communicate with container, so can nodemanager directly communicate with containers which are running on different nodes or it will go through RM to get container information.

Thank you.

[#3 · Accepted solution]

Hi @Gunjan Dhawas

For point 2 - Also as you mentioned its nodemanager which will communicate with container, so can nodemanager directly communicate with containers which are running on different nodes or it will go through RM to get container information.

Nodemanagers are basically YARN’s per-node agent, and takes care of the individual compute nodes in a Hadoop cluster. This includes keeping up-to date with the ResourceManager (RM), overseeing containers’ life-cycle management; monitoring resource usage (memory, CPU) of individual containers, tracking node-health, log’s management and auxiliary services which may be exploited by different YARN applications.

 So Nodemanagers are the nodes on which containers are launched. So yes, nodemanagers directly monitors the containers and their resource consumption.

For point 1 - "The application code executing within the container then provides necessary information (progress, status etc.) to its ApplicationMaster via an application-specific protocol.", so how the application master monitor the status of containers which are running on different node than applicatioMaster.

Once the applicationMaster negotiates the resources with RM, it will launch the container by providing container launch specification to the NodeManager.  The launch specification includes the necessary information to allow the container to communicate with the ApplicationMaster itself. Thus ApplicationMaster gets the progress/status via application-specific protocol provided in the container launch specification