Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Node labels for Map Reduce jobs

Highlighted

Node labels for Map Reduce jobs

As per this link https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_yarn-resource-management/content/using_n... Node labels are not supported for map reduce jobs. Are there any plans of supporting Node Labels for Map Reduce jobs?

2 REPLIES 2
Highlighted

Re: Node labels for Map Reduce jobs

If you keep on reading that paragraph, it says "Currently you cannot specify a node label when submitting a MapReduce job. However, if you submit a MapReduce job to a queue that has a default node label expression, the default node label will be applied to the MapReduce job." So, assign a Yarn label, say "y1" to a few nodes, create a Yarn queue, say "q1" and set its "default node label expression" to "y1", using for example Yarn Queue Manager Ambari view. Then, you can run a MR job on nodes with that label by setting the job's queue, like this:

$ hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar pi -D mapreduce.job.queuename=q1 10 10000

Setting queue name is the way to go, not using labels directly. Setting Yarn queue is supported by all services that can run on Yarn, like Hive, Pig, Oozie, Spark etc.

Highlighted

Re: Node labels for Map Reduce jobs

Thank you for your response. I understand that we can use queues. But here I am trying to create Nodes on demand and assigning the labels (exclusive=true) to those nodes, so that my job runs on specified nodes. If I create Queues on demand, I need to change / recalculate capacity for all the existing queues.(Not sure if there an easy way to do this) That is the reason I want to go with the same Queue but with different node labels.

Don't have an account?
Coming from Hortonworks? Activate your account here