My nodemanger deployed on host with 1TB memory
while YARN memory 100% running
From centos , the free memory still have more than 300GB. Why?
Hello! The YARN memory allocated is based on what the ApplicationMaster has requested for. The OS free memory is based on what's actually being used.
For example, MR can ask for a 4GB container for a mapper, which actually uses only 2GB. In this case allocated memory will be 4GB and the OS will show 2GB free.
Thansk for @vvasudev quick response. So how can I make the physical memory usage more efficient? It seems not all physical memory used for the yarn containers. Is my yarn memory configuration reasonable?
Your memory settings are reasonable. The only way to improve the memory usage is -
1. To run more containers.
2. Do more work in a single container.
It's probably easier to do (1).