@Roberto Ayuso In spark, spark.driver.memoryOverhead is considered in calculating the total memory required for the driver. By default it is 0.10 of the driver-memory or minimum 384MB. In your case it will be 8GB * 0.1 = 9011MB ~= 9G YARN allocates memory only in increments/multiples of yarn.scheduler.minimum-allocation-mb . When yarn.scheduler.minimum-allocation-mb=4G, it can only allocate container sizes of 4G,8G,12G etc. So if something like 9G is requested it will round up to the next multiple and will allocate 12G of container size for the driver. When yarn.scheduler.minimum-allocation-mb=1G, then container sizes of 8G, 9G, 10G are possible. The nearest rounded up size of 9G will be used in this case. ... View more

@Amila Silva HDP 3.0 supports GPU isolation in docker using nvidia-docker-plugin https://github.com/NVIDIA/nvidia-docker/wiki/nvidia-docker-plugin which is part of nvidia-docker v1. Currently only this is supported and not the newer version. ... View more

@Michael Bronson I will look to create an article about configuring the vcores for cpu scheduling when I get time. I will mention this part there. ... View more

Interesting. Can you paste the lscpu output of the nodes you are mentioning? ... View more

@Michael Bronson Yarn Vcores can ideally be set up to 2x the actual cpu present based on the use case. Thats why ambari provides the option in the scroll bar. It does not depend on the number of threads shown in lscpu. If you want to prevent over utilization of cpu by Yarn and leave cpu for OS and other processes you can set to 80% of 32 . But keep in mind that this value will only be considered by YARN for scheduling containers, when CPU scheduling is enabled. ... View more

@Michael Bronson The lscpu output of "CPU(s):" itself always takes into consideration the "Thread(s) per core". So usually, CPU(s) = [Thread(s) per core] X [Core(s) per socket] X [Socket(s)] It is sufficient to consider only CPU(s), when setting yarn.nodemanager.resource.cpu-vcores . ... View more

@naveen r The amount of resource a Yarn application will request is completely dependent on the type of Application. For MapReduce, it is generally based on the inputsplit/no. of reducers configured and the memory/vcores configured per mapper/reducer in the JobConf. To check how much Resources are currently being used by a single application, you can use the following Rest API and check the fields "allocatedMB","allocatedVCores" and "runningContainers" GET http://<rm http address:port>/ws/v1/cluster/apps/<applicationId> ... View more