Getting a physical machines with all its cores from YARN

[#1]

Is it possible to ask YARN to allocate one full physical machine (with all of its cores) to one application? E.g. If I need 24 cores and there is a machine in the cluster which has 24 cores, I would like to get that full physical box instead of 24 vcores which are potentially scattered in the cluster. Or if I have apriori knowledge that in my cluster no machine has less than n cores, I would like YARN to give me x machines (with all its cores) instead of asking in terms of vcores.

[#2 · Accepted solution]

@Abdul Qadeer

I see what are you asking. Basically you want your application to be run on dedicated nodemanager. You can have a look at YARN's node label feature.

https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/NodeLabel.html

Please do let me know if this is not what you are looking for.

[#3]

@Kuldeep Kulkarni

Thank you for giving me a good starting point. Let me do some experimentation using YARN's node labels and then I might have some follow-up questions.

[#4]

@Abdul Qadeer

You are welcome Sir! If this helped you, you can accept my answer and post a new question whenever you need our help.