Hive Tez Client Memory

gbrahmi — Wed, 19 Jul 2017 05:00:06 GMT

I am trying to follow a strange problem I am seeing with Hive on Tez on the sandbox to make things simple.

When I run hive cli I see there is a container occupied in YARN via Application for the session. It however is using 500 MB of memory. I am trying to figure out where is this number coming up. The HADOOP_HEAPSIZE is set at 250MB.

The minimum container size in YARN is yarn.scheduler.minimum-allocation-mb is 250. The Tez container size in hive-site is set at 250 (hive.tez.container.size).

Even after changing hive.tez.container.size I don't see any change in the memory utilization for the single YARN Application container.

I've also tried to play around with the tez.am.resource.memory.mb and tez.dag.am.resource.memory.mb but nothing worked.

Am I missing something? Is there some kind of calculation with the HADOOP_HEAPSIZE which should be affecting the number to change?

I am trying to see if the idle session Hive cli connection memory can be brought down since they keep the resources hooked up on a small cluster.

Re: Hive Tez Client Memory

rajkumar_singh — Wed, 19 Jul 2017 09:43:27 GMT

@Gagan Brahmi modify tez.am.resource.memory.mb in ambari tez configuration, restart tez and hive client.

Re: Hive Tez Client Memory

gbrahmi — Thu, 20 Jul 2017 07:23:34 GMT

Strangely when I tried to modify the tez.am.resource.memory.mb under Tez yesterday it wasn't taking effect. However, it does seem to be taking effect now. Appreciate your inputs.

question Hive Tez Client Memory in Archives of Support Questions (Read Only)

Hive Tez Client Memory

Re: Hive Tez Client Memory

Re: Hive Tez Client Memory