I am tuning our Yarn cluster which hosts tez, mapreduce and Spark-on-Yarn. I observed during busy hour, all Yarn cluster memory is used up while lots of cores are free--- which leads me to believe that we should decrease minimum container size. Meanwhile, lots of applications specify their memory, -xms=4GB etc; considering container overhead memory, would not it requires container size bigger than 4G?
If I set yarn.scheduler.minimum-allocation-mb = 4GB, Spark executor size is 4GB as well, would it actually assign 2 containers/8GB to the Spark executor or yarn is smart enough to allocate around 5GB? Even in HWX tutorial, I think something not right: In https://community.hortonworks.com/content/kbentry/14309/demystify-tez-tuning-step-by-step.html hive.tez.container.size is multiple of yarn.scheduler.minimum-allocation-mb, why so? if yarn.scheduler.maximum-allocation-mb = 24GB, yarn.scheduler.minimum-allocation-mb = 4GB, hive.tez.container.size=5B, would not Yarn smart enough to assign 5GB to a container to satisfy tez needs?
... View more
