08-12-2015 10:40 PM
I have a 2 node cluster with each node having 8GB RAM and 4 cores.
On both node, there are apps running having consumed 2 cores each. This leaves me with 2 cores (x 2) on both nodes. Memory used is 4GB of total 16GB available to YARN containers.
Some important properties:
yarn.nodemanager.resource.memory-mb = 20GB (overcomitted, as I see)
yarn.scheduler.minimum-allocation-mb = 1GB
yarn.scheduler.maximum-allocation-mb = 5.47GB
yarn.nodemanager.resource.cpu-vcores = 12
yarn.scheduler.minimum-allocation-vcores = 1
yarn.scheduler.maximum-allocation-vcores = 12
Using Fair scheduler.
With above setting. when I spark-submit the app remains in ACCEPTED state. Here is what I am requesting through this:
num-executors = 2
executor-memory = 1G
executor-cores = 1
As I see, I am requesting a total of 3 cores (1 for driver, by default and 1x2 for executors). A single node does not have 3 cores but has 2 cores. So, ideally I should see them distributed across 2 nodes. Not sure why the spark job should remain in ACCEPTED state. My default queue shows only 25% usage.
I notice the following settings too for my default root.default queue:
Used Capacity: 25.0%
Used Resources: <memory:4096, vCores:4>
Num Schedulable Applications: 2
Num Non-Schedulable Applications: 1
Num Containers: 4
Max Schedulable Applications: 2
Max Schedulable Applications Per User: 2
Why do I only get 4 containers in total? Or does it this indicate currently used containers (which in my case is 4)?
Plus, why is max schedulable apps only 2? I have not set any user level limits or queue level limits under Dynamic Resource Pool settings.
08-18-2015 11:58 PM
If you run spark on yarn you have two modes: client and cluster. If you run in cluster mode the AM also runs the driver. If you run client mode the driver runs locally and there is just an AM to manage the executors. Check running on yarn for the full details.
The amount of resources that can be used for AM's is limited, check for the "Max Application Master Share" for the queues.
On top of the memory requirement that you have set for the executor are the heap sizes on top of that the overhead will be added (7% with a minimum of 384MB). That will increase you container request and could see it being round up to 2GB based on the increment value you have set. Check yarn.scheduler.increment-allocation-mb
Like you did you need to check what is available on each node to see if you have room, having space in the cluster does not always mean that you can satisfy the request.
08-20-2015 04:20 PM
The setting is part of the queue in the dynamic resource pools configuration for Cloudera Manager.
It is exposed in CM 5.4 and is only available from CDH 5.1 onwards.
The scheduler does not look at disks as a resource (it might in the future) but for now follow the YARN tuning documentation which will take into account the disks when you calculate the values.
08-20-2015 08:22 PM
So, the only thing which is not clear to me now is why was I allowed only 2 max apps per user when it was not set anywhere explicitly? Is it possible to infer that from data being given above? Or what other data point(s) influenced limiting it to 2 apps per user?
08-25-2015 08:08 AM
I notice that memory on another CDH host is overcommitted.
Of the total 62GB phy memory, the memory allotted to YARN containers is 40GB and that consumes the major chunk. I notice that my apps remain in ACCEPTED state despite requesting for only 1GB for each container.
1. Is this 40GB reserved by YARN at startup?
2. Plus, I assume that does not contribute to apps remaining in ACCEPTED state because it is from this pool of 40GB that my apps should get memory allocated, right?