Hive Tez Client Memory

[#1]

I am trying to follow a strange problem I am seeing with Hive on Tez on the sandbox to make things simple.

When I run hive cli I see there is a container occupied in YARN via Application for the session. It however is using 500 MB of memory. I am trying to figure out where is this number coming up. The HADOOP_HEAPSIZE is set at 250MB.

The minimum container size in YARN is yarn.scheduler.minimum-allocation-mb is 250. The Tez container size in hive-site is set at 250 (hive.tez.container.size).

Even after changing hive.tez.container.size I don't see any change in the memory utilization for the single YARN Application container.

I've also tried to play around with the tez.am.resource.memory.mb and tez.dag.am.resource.memory.mb but nothing worked.

Am I missing something? Is there some kind of calculation with the HADOOP_HEAPSIZE which should be affecting the number to change?

I am trying to see if the idle session Hive cli connection memory can be brought down since they keep the resources hooked up on a small cluster.

[#2 · Accepted solution]

@Gagan Brahmi modify tez.am.resource.memory.mb in ambari tez configuration, restart tez and hive client.

[#3]

Strangely when I tried to modify the tez.am.resource.memory.mb under Tez yesterday it wasn't taking effect. However, it does seem to be taking effect now. Appreciate your inputs.