Created 02-17-2018 09:38 AM
can we run more no of containers than the available cpu core? if yes then how can we achieve it?
As per my understanding we can run max no of containers as per the value configured in yarn i.e yarn.scheduler.minimum allocation and yarn.scheduler.maximum allocation.
But , suppose in my cluster total cpu core or vcore is 100 and i want to run the 500 containers simultaneously. Then how can i achieve the same?
Created 02-23-2018 06:23 AM
By Default CPU is shared unless you enable CPU based scheduling. So you will be able to run (total yarn memory/container size) on your cluster.
Created 03-02-2018 04:16 AM
@Pranay Vyas So if we have 400GB yarn memory and only 10 core of cpu. Then can we run 400 containers simultaneously in a cluster?using minimum.allocation = 1GB
Created 04-10-2018 03:15 PM
Not really. you also need AM containers to spawn new AM for your jobs. You can configure % of memory reserved for AM containers in Yarn capacity scheduler view.