Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Max apps per user.

SOLVED Go to solution

Max apps per user.

Contributor

I have a 2 node cluster with each node having 8GB RAM and 4 cores

 

On both node, there are apps running having consumed 2 cores each. This leaves me with 2 cores (x 2) on both nodes. Memory used is 4GB of total 16GB available to YARN containers.

 

Some important properties:

yarn.nodemanager.resource.memory-mb = 20GB   (overcomitted, as I see)

yarn.scheduler.minimum-allocation-mb = 1GB

yarn.scheduler.maximum-allocation-mb = 5.47GB

 

yarn.nodemanager.resource.cpu-vcores = 12

yarn.scheduler.minimum-allocation-vcores = 1

yarn.scheduler.maximum-allocation-vcores = 12

 

Using Fair scheduler.

 

With above setting. when I spark-submit the app remains in ACCEPTED state. Here is what I am requesting through this:

 

spark.driver.memory=2G 

spark.master=yarn-client

spark.executor.memory=1G

num-executors = 2

executor-memory = 1G

executor-cores = 1

 

As I see, I am requesting a total of 3 cores (1 for driver, by default and 1x2 for executors). A single node does not have 3 cores but has 2 cores. So, ideally I should see them distributed across 2 nodes. Not sure why the spark job should remain in ACCEPTED state.  My default queue shows only 25% usage.

 

I notice the following settings too for my default root.default queue:

Used Capacity:                                          25.0%
Used Resources:                                        <memory:4096, vCores:4>
Num Schedulable Applications:                  2
Num Non-Schedulable Applications:          1
Num Containers:                                         4
Max Schedulable Applications:                   2
Max Schedulable Applications Per User:    2

 

Why do I only get 4 containers in total? Or does it this indicate currently used containers (which in my case is 4)?

Plus, why is max schedulable apps only 2? I have not set any user level limits or queue level limits under Dynamic Resource Pool settings. 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Max apps per user.

Super Collaborator

The setting is part of the queue in the dynamic resource pools configuration for Cloudera Manager.

It is exposed in CM 5.4 and is only available from CDH 5.1 onwards.

The scheduler does not look at disks as a resource (it might in the future) but for now follow the YARN tuning documentation which will take into account the disks when you calculate the values.

 

Wilfred

7 REPLIES 7

Re: Max apps per user.

Contributor

Ok, figured one thing that Num Containers implies total running containers.

 

I have 2 independent physical disks on both nodes.

Re: Max apps per user.

Super Collaborator

If you run spark on yarn you have two modes: client and cluster. If you run in cluster mode the AM also runs the driver. If you run client mode the driver runs locally and there is just an AM to manage the executors. Check running on yarn for the full details.

The amount of resources that can be used for AM's is limited, check for the "Max Application Master Share" for the queues.

 

On top of the memory requirement that you have set for the executor are the heap sizes on top of that the overhead will be added (7% with a minimum of 384MB). That will increase you container request and could see it being round up to 2GB based on the increment value you have set. Check yarn.scheduler.increment-allocation-mb

 

Like you did you need to check what is available on each node to see if you have room, having space in the cluster does not always mean that you can satisfy the request.

 

Wilfred

Re: Max apps per user.

Contributor

Thank you Wilfred.

 

Where do I check "Max Application Master Share"?

 

Plus, does Fair Scheduler take independent disks into account when calculating number of containers possible? 

Re: Max apps per user.

Super Collaborator

The setting is part of the queue in the dynamic resource pools configuration for Cloudera Manager.

It is exposed in CM 5.4 and is only available from CDH 5.1 onwards.

The scheduler does not look at disks as a resource (it might in the future) but for now follow the YARN tuning documentation which will take into account the disks when you calculate the values.

 

Wilfred

Re: Max apps per user.

Contributor

So, the only thing which is not clear to me now is why was I allowed only 2 max apps per user when it was not set anywhere explicitly? Is it possible to infer that from data being given above? Or what other data point(s) influenced limiting it to 2 apps per user?

 

Thanks,

Sumit

Highlighted

Re: Max apps per user.

Super Collaborator

No sorry I can not deduce that from the data given.

Check the RM logs that is whee all scheduling activities are logged.

 

Wilfred

Re: Max apps per user.

Contributor

I notice that memory on another CDH host is overcommitted.

 

Of the total 62GB phy memory, the memory allotted to YARN containers is 40GB and that consumes the major chunk. I notice that my apps remain in ACCEPTED state despite requesting for only 1GB for each container. 

 

1. Is this 40GB reserved by YARN at startup?

2. Plus, I assume that does not contribute to apps remaining in ACCEPTED state because it is from this pool of 40GB that my apps should get memory allocated, right?

 

Thanks,

Sumit