question Re: Getting a physical machines with all its cores from YARN in Archives of Support Questions (Read Only)

Getting a physical machines with all its cores from YARN

qadeer_qadeer — Wed, 03 Aug 2016 02:47:03 GMT

Is it possible to ask YARN to allocate one full physical machine (with all of its cores) to one application? E.g. If I need 24 cores and there is a machine in the cluster which has 24 cores, I would like to get that full physical box instead of 24 vcores which are potentially scattered in the cluster. Or if I have apriori knowledge that in my cluster no machine has less than n cores, I would like YARN to give me x machines (with all its cores) instead of asking in terms of vcores.

Re: Getting a physical machines with all its cores from YARN

KuldeepK — Wed, 03 Aug 2016 05:54:05 GMT

@Abdul Qadeer

I see what are you asking. Basically you want your application to be run on dedicated nodemanager. You can have a look at YARN's node label feature.

https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/NodeLabel.html

Please do let me know if this is not what you are looking for.

Re: Getting a physical machines with all its cores from YARN

qadeer_qadeer — Thu, 04 Aug 2016 01:30:34 GMT

@Kuldeep Kulkarni

Thank you for giving me a good starting point. Let me do some experimentation using YARN's node labels and then I might have some follow-up questions.

Re: Getting a physical machines with all its cores from YARN

KuldeepK — Thu, 04 Aug 2016 01:34:53 GMT

@Abdul Qadeer

You are welcome Sir! If this helped you, you can accept my answer and post a new question whenever you need our help.