question Re: Is it possible to assign Hadoop queues to Hadoop tools ? in Archives of Support Questions (Read Only)

Is it possible to assign Hadoop queues to Hadoop tools ?

MFP — Fri, 16 Sep 2022 12:02:58 GMT

Is it possible to assign Hadoop queues to Hadoop tools via Capacity scheduler ?

Example :

Hive queue will only cater request from Hive clients.

Spark queue will only cater request from Spark clients.

Question in another way :

Is it possible to map Hadoop components to Hadoop queue ?

Regards,

Fahim

Re: Is it possible to assign Hadoop queues to Hadoop tools ?

mqureshi — Fri, 04 Aug 2017 21:00:57 GMT

@Mohammedfahim Pathan

You assign queues when you run your jobs. For example in spark you can specify --queue parameter. In YARN configuration you specify ACL on who is part of the queue. This can be users that are running those "tools". So you cannot say that a queue is meant for "Hive", but when you limit a queue to a group using Hive only, then in a way you achieve your purpose.

 <property>
   <name>yarn.scheduler.capacity.queue-mappings</name>
   <value>u:user1:queue1,g:group1:queue2,u:%user:%user,u:user2:%primary_group</value>
   <description>
     Here, <user1> is mapped to <queue1>, <group1> is mapped to <queue2>, 
     maps users to queues with the same name as user, <user2> is mapped 
     to queue name same as <primary group> respectively. The mappings will be 
     evaluated from left to right, and the first valid mapping will be used.
   </description>
 </property>

Re: Is it possible to assign Hadoop queues to Hadoop tools ?

MFP — Tue, 08 Aug 2017 18:55:34 GMT

@mqureshi , Thanks for answer. It makes sense.

Regards,

Fahim