question Re: Tez out of memory? in Archives of Support Questions (Read Only)

Tez out of memory?

honza_nguyen1 — Sun, 18 Aug 2019 07:02:43 GMT

Hi guys,

I am trying to execute a command from zeppelin. I use the sandbox 2.6 and my VM has 16GB ram.

I dont know why it is out of memmory. Could anybody help me?

Re: Tez out of memory?

jsensharma — Fri, 10 Nov 2017 18:32:59 GMT

@enzo EL

Where do you see the OutOfMemory ? Can you please share the complete trace of OutOfMemory and where do you see it.

What is your "hive.tez.container.size" value ? (Can you try increasing it and then see if it helps).

If the Hive Table is ORC formatted then you can also refer to the following article for tuning: https://community.hortonworks.com/articles/68631/optimizing-hive-queries-for-orc-formatted-tables.html

Re: Tez out of memory?

honza_nguyen1 — Sun, 18 Aug 2019 07:02:36 GMT

oh sorry...this will be better. Where could I foung info abou "hive.tez.container.size" value ?

Thanks

Re: Tez out of memory?

jsensharma — Sun, 18 Aug 2019 07:02:28 GMT

@enzo EL

Ambari UI --> Hive --> Configs --> Settings Tab

hive.tez.container.size

Re: Tez out of memory?

honza_nguyen1 — Fri, 10 Nov 2017 21:24:32 GMT

No I set the value to 900MB and it is still not working...

Re: Tez out of memory?

jsensharma — Fri, 10 Nov 2017 21:27:05 GMT

@enzo EL

Thats a tuning parameter .. so it's value depends on your requirement or the kind of job that you are running.

Please try to set it to a higher value like 2500MB and then see if it works.

Re: Tez out of memory?

honza_nguyen1 — Sun, 18 Aug 2019 07:02:20 GMT

I set it up....but i dont know, why my easy query select count (*) from cdr use 90% of Yarn ..?

take a look at screenshots..

Re: Tez out of memory?

jsensharma — Fri, 10 Nov 2017 22:00:45 GMT

@enzo EL

Please also try to setup the following parameters:

set hive.tez.java.opts=-Xmx1640m (0.8 times hive.tez.container.size)
set tez.runtime.io.sort.mb=820 (0.4 times hive.tez.container.size)
set tez.runtime.unordered.output.buffer.size-mb=205 (0.1 times hive.tez.container.size)

The following link provides good examples and explaination on Tez Tuning.
https://community.hortonworks.com/articles/14309/demystify-tez-tuning-step-by-step.html

Re: Tez out of memory?

honza_nguyen1 — Sat, 11 Nov 2017 18:28:51 GMT

Thank you....

According to your posted links and this link https://community.hortonworks.com/questions/67677/javalangoutofmemoryerror-java-heap-space-error.html I solved the problem.

Thank you.

Re: Tez out of memory?

jsensharma — Sat, 11 Nov 2017 18:38:38 GMT

@enzo EL

Good to know that the issue is resolved. As the issue is resolved, hence it will be also great if you can mark this HCC thread as Answered by clicking on the "Accept" Button on the correct answer. That way other HCC users can quickly find the solution when they encounter the same issue.