In cluster metrics, memory used is always equal to number of vCores in use.
Forex: if 18 vcores are in use, memory used is 18 GB.This looks weird to me. Could someone please explain what's happening here , if I should allow more memory to be used and how to do that
I understand by default 1 gb is allocated to each of the containers and number of containers running at a time is equal to vcores used.
However, I see Memory available in my cluster is 150GB and being used it upto 50 GB since that is the number of vcores.
I am getting these numbers from cluster metrics.
metrics for referrence:
Yarn configuration provides a min and max container size for memory. If you like to use more per container you should increase the container min size for memory.
For example, if you set min container size memory to 2G, then when all containers are running memory utilization will grow to 100G instead of just 50G.