Max apps per user.

[#1]

I have a 2 node cluster with each node having 8GB RAM and 4 cores. 

On both node, there are apps running having consumed 2 cores each. This leaves me with 2 cores (x 2) on both nodes. Memory used is 4GB of total 16GB available to YARN containers.

Some important properties:

yarn.nodemanager.resource.memory-mb = 20GB   (overcomitted, as I see)

yarn.scheduler.minimum-allocation-mb = 1GB

yarn.scheduler.maximum-allocation-mb = 5.47GB

yarn.nodemanager.resource.cpu-vcores = 12

yarn.scheduler.minimum-allocation-vcores = 1

yarn.scheduler.maximum-allocation-vcores = 12

Using Fair scheduler.

With above setting. when I spark-submit the app remains in ACCEPTED state. Here is what I am requesting through this:

spark.driver.memory=2G 

spark.master=yarn-client

spark.executor.memory=1G

num-executors = 2

executor-memory = 1G

executor-cores = 1

As I see, I am requesting a total of 3 cores (1 for driver, by default and 1x2 for executors). A single node does not have 3 cores but has 2 cores. So, ideally I should see them distributed across 2 nodes. Not sure why the spark job should remain in ACCEPTED state.  My default queue shows only 25% usage.

I notice the following settings too for my default root.default queue:

Used Capacity:                                          25.0%
Used Resources:                                        <memory:4096, vCores:4>
Num Schedulable Applications:                  2
Num Non-Schedulable Applications:          1
Num Containers:                                         4
Max Schedulable Applications:                   2
Max Schedulable Applications Per User:    2

Why do I only get 4 containers in total? Or does it this indicate currently used containers (which in my case is 4)?

Plus, why is max schedulable apps only 2? I have not set any user level limits or queue level limits under Dynamic Resource Pool settings. 

[#2]

Ok, figured one thing that Num Containers implies total running containers.

I have 2 independent physical disks on both nodes.

[#3]

If you run spark on yarn you have two modes: client and cluster. If you run in cluster mode the AM also runs the driver. If you run client mode the driver runs locally and there is just an AM to manage the executors. Check running on yarn for the full details.

The amount of resources that can be used for AM's is limited, check for the "Max Application Master Share" for the queues.

On top of the memory requirement that you have set for the executor are the heap sizes on top of that the overhead will be added (7% with a minimum of 384MB). That will increase you container request and could see it being round up to 2GB based on the increment value you have set. Check yarn.scheduler.increment-allocation-mb

Like you did you need to check what is available on each node to see if you have room, having space in the cluster does not always mean that you can satisfy the request.

Wilfred

[#4]

Thank you Wilfred.

Where do I check "Max Application Master Share"?

Plus, does Fair Scheduler take independent disks into account when calculating number of containers possible? 

[#5 · Accepted solution]

The setting is part of the queue in the dynamic resource pools configuration for Cloudera Manager.

It is exposed in CM 5.4 and is only available from CDH 5.1 onwards.

The scheduler does not look at disks as a resource (it might in the future) but for now follow the YARN tuning documentation which will take into account the disks when you calculate the values.

Wilfred

[#6]

So, the only thing which is not clear to me now is why was I allowed only 2 max apps per user when it was not set anywhere explicitly? Is it possible to infer that from data being given above? Or what other data point(s) influenced limiting it to 2 apps per user?

Thanks,

Sumit

[#7]

No sorry I can not deduce that from the data given.

Check the RM logs that is whee all scheduling activities are logged.

Wilfred

[#8]

I notice that memory on another CDH host is overcommitted.

Of the total 62GB phy memory, the memory allotted to YARN containers is 40GB and that consumes the major chunk. I notice that my apps remain in ACCEPTED state despite requesting for only 1GB for each container. 

1. Is this 40GB reserved by YARN at startup?

2. Plus, I assume that does not contribute to apps remaining in ACCEPTED state because it is from this pool of 40GB that my apps should get memory allocated, right?

Thanks,

Sumit