Support Questions
Find answers, ask questions, and share your expertise

can we run more no of containers than the available cpu core? if yes then how can we achieve it?

can we run more no of containers than the available cpu core? if yes then how can we achieve it?

Explorer

can we run more no of containers than the available cpu core? if yes then how can we achieve it?

As per my understanding we can run max no of containers as per the value configured in yarn i.e yarn.scheduler.minimum allocation and yarn.scheduler.maximum allocation.

But , suppose in my cluster total cpu core or vcore is 100 and i want to run the 500 containers simultaneously. Then how can i achieve the same?

3 REPLIES 3
Highlighted

Re: can we run more no of containers than the available cpu core? if yes then how can we achieve it?

Expert Contributor

By Default CPU is shared unless you enable CPU based scheduling. So you will be able to run (total yarn memory/container size) on your cluster.

Highlighted

Re: can we run more no of containers than the available cpu core? if yes then how can we achieve it?

Explorer

@Pranay Vyas So if we have 400GB yarn memory and only 10 core of cpu. Then can we run 400 containers simultaneously in a cluster?using minimum.allocation = 1GB

Highlighted

Re: can we run more no of containers than the available cpu core? if yes then how can we achieve it?

Expert Contributor

Not really. you also need AM containers to spawn new AM for your jobs. You can configure % of memory reserved for AM containers in Yarn capacity scheduler view.