Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Why is memory used in cluster equal to vCores used

Why is memory used in cluster equal to vCores used

Expert Contributor

In cluster metrics, memory used is always equal to number of vCores in use.

Forex: if 18 vcores are in use, memory used is 18 GB.This looks weird to me. Could someone please explain what's happening here , if I should allow more memory to be used and how to do that

I understand by default 1 gb is allocated to each of the containers and number of containers running at a time is equal to vcores used.

However, I see Memory available in my cluster is 150GB and being used it upto 50 GB since that is the number of vcores.

I am getting these numbers from cluster metrics.

metrics for referrence:

Cluster Metrics

Apps SubmittedApps PendingApps RunningApps CompletedContainers RunningMemory UsedMemory TotalMemory ReservedVCores UsedVCores TotalVCores Reserved
21400921312424 GB150 GB0 B24560

Re: Why is memory used in cluster equal to vCores used

@Simran Kaur

In your case each container is using 1G and 1 vcore this is why the Memory Used = VCores Used.

Yarn configuration provides a min and max container size for memory. If you like to use more per container you should increase the container min size for memory.

yarn.scheduler.minimum-allocation-mb=1024 (default)

yarn.scheduler.maximum-allocation-mb=8192 (default)

For example, if you set min container size memory to 2G, then when all containers are running memory utilization will grow to 100G instead of just 50G.


Don't have an account?
Coming from Hortonworks? Activate your account here