Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Yarn number of container allocation

Highlighted

Yarn number of container allocation

New Contributor

Hi Team,

I have read that AM in yarn asks for no of containers based on the current availability of resources. So my question is

Case 1 - If I run my job when no other job is running in the same queue then my job will get more no of containers.

Case 2- If queue is already occupied then job will get less no of containers.

Case 3- My job will get fixed no of containers.

It would be greatly helpful if you can tell my understanding is correct or not and on which parameter AM master asks for no of containers from RM.

2 REPLIES 2

Re: Yarn number of container allocation

Cloudera Employee

@Gopi Sharma

Number of containers each query will use is defined here (https://cwiki.apache.org/confluence/display/TEZ/How+initial+task+parallelism+works), which consider number of resource available on current queue, the number of resource available in a queue is defined by the minimum guaranteed capacity (yarn.scheduler.capacity.root._queuename_.capacity) and not maximum capacity (yarn.scheduler.capacity.root._queuename_.maximum-capacity).

You can read this https://community.hortonworks.com/articles/56636/hive-understanding-concurrent-sessions-queue-alloc....

Re: Yarn number of container allocation

New Contributor

@Gopi Sharma

YARN ResourceManager does its best to allocate memory and vcores to use all available resources in the most efficient way possible ideally there will be some or few resources left idle