Is it possible to assign Hadoop queues to Hadoop tools ?

[#1]

Is it possible to assign Hadoop queues to Hadoop tools via Capacity scheduler ?

Example :

Hive queue will only cater request from Hive clients.

Spark queue will only cater request from Spark clients.

Question in another way :

Is it possible to map Hadoop components to Hadoop queue ?

Regards,

Fahim

[#2 · Accepted solution]

@Mohammedfahim Pathan

You assign queues when you run your jobs. For example in spark you can specify --queue parameter. In YARN configuration you specify ACL on who is part of the queue. This can be users that are running those "tools". So you cannot say that a queue is meant for "Hive", but when you limit a queue to a group using Hive only, then in a way you achieve your purpose.

 <property>
   <name>yarn.scheduler.capacity.queue-mappings</name>
   <value>u:user1:queue1,g:group1:queue2,u:%user:%user,u:user2:%primary_group</value>
   <description>
     Here, <user1> is mapped to <queue1>, <group1> is mapped to <queue2>, 
     maps users to queues with the same name as user, <user2> is mapped 
     to queue name same as <primary group> respectively. The mappings will be 
     evaluated from left to right, and the first valid mapping will be used.
   </description>
 </property>

[#3]

@mqureshi , Thanks for answer. It makes sense.

Regards,

Fahim