I am trying to set hive.tez.container.size but it is not taking effect, I am seeing a big task container getting launched, I have also tried setting tez.task.resource.memory.mb but no luck.please let me know how to set it properly

[#1 · Accepted solution]

what is yarn.scheduler.minimum-allocation-mb set to? in your case it looks yarn.scheduler.minimum-allocation-mb is set too high and you are trying to set hive.tez.container.size lower than this value. if this is the case then you will see your task container get launched with the yarn.scheduler.minimum-allocation-mb.

the best practice is to set hive.tez.container.size to be the same as or a small multiple (1 or 2 times that) of YARN container size yarn.scheduler.minimum-allocation-mb but never more than yarn.scheduler.maximum-allocation-mb. You want to have headroom for multiple containers to be spun up.