Reply
Contributor
Posts: 55
Registered: ‎09-17-2013

How does Application master knows resource name (node name/rack name).?

As per YARN specifications, A resource request from Application master will have following properties
1) Desired Resource Name (host name, rack name or * for no preference)
2) Priority
3) magnitude/requirements
4) No of containers

 

My question here is, how does AM knows the Rack or Node name even before allocating the container.?
Will it first reach Resource Manager and gets the Node’s/Block’s information .? and then requests for container to be launched in that node.?

 

 

Posts: 1,892
Kudos: 431
Solutions: 302
Registered: ‎07-31-2013

Re: How does Application master knows resource name (node name/rack name).?

Yes, you could gather this info at the client-side before the AM is triggered. That is how we do it in MR2, for example.

The InputSplit created at the client-end [1] carry the HDFS-provided info about the host and rack, which we end up using later in the submitted MR AM to schedule the actual containers [2].

[1] - https://github.com/cloudera/hadoop-common/blob/cdh5.4.0-release/hadoop-mapreduce-project/hadoop-mapr... and https://github.com/cloudera/hadoop-common/blob/cdh5.4.0-release/hadoop-mapreduce1-project/src/mapred...
[2] - https://github.com/cloudera/hadoop-common/blob/cdh5.4.0-release/hadoop-mapreduce-project/hadoop-mapr... and https://github.com/cloudera/hadoop-common/blob/cdh5.4.0-release/hadoop-mapreduce-project/hadoop-mapr...
Announcements