Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

can we run more no of containers than the available cpu core? if yes then how can we achieve it?

Highlighted

can we run more no of containers than the available cpu core? if yes then how can we achieve it?

New Contributor

can we run more no of containers than the available cpu core? if yes then how can we achieve it?

As per my understanding we can run max no of containers as per the value configured in yarn i.e yarn.scheduler.minimum allocation and yarn.scheduler.maximum allocation.

But , suppose in my cluster total cpu core or vcore is 100 and i want to run the 500 containers simultaneously. Then how can i achieve the same?

3 REPLIES 3

Re: can we run more no of containers than the available cpu core? if yes then how can we achieve it?

Expert Contributor

By Default CPU is shared unless you enable CPU based scheduling. So you will be able to run (total yarn memory/container size) on your cluster.

Re: can we run more no of containers than the available cpu core? if yes then how can we achieve it?

New Contributor

@Pranay Vyas So if we have 400GB yarn memory and only 10 core of cpu. Then can we run 400 containers simultaneously in a cluster?using minimum.allocation = 1GB

Re: can we run more no of containers than the available cpu core? if yes then how can we achieve it?

Expert Contributor

Not really. you also need AM containers to spawn new AM for your jobs. You can configure % of memory reserved for AM containers in Yarn capacity scheduler view.