Tez out of memory?

[#1]

Hi guys,

I am trying to execute a command from zeppelin. I use the sandbox 2.6 and my VM has 16GB ram.

I dont know why it is out of memmory. Could anybody help me?

[#2]

@enzo EL

Where do you see the OutOfMemory ? Can you please share the complete trace of OutOfMemory and where do you see it.

What is your "hive.tez.container.size" value ? (Can you try increasing it and then see if it helps).

If the Hive Table is ORC formatted then you can also refer to the following article for tuning: https://community.hortonworks.com/articles/68631/optimizing-hive-queries-for-orc-formatted-tables.ht...

.

[#3]

oh sorry...this will be better. Where could I foung info abou "hive.tez.container.size" value ?

Thanks

[#4]

@enzo EL

Ambari UI --> Hive --> Configs --> Settings Tab

hive.tez.container.size

.

[#5]

No I set the value to 900MB and it is still not working...

[#6]

@enzo EL

Thats a tuning parameter .. so it's value depends on your requirement or the kind of job that you are running.

Please try to set it to a higher value like 2500MB and then see if it works.

[#7]

I set it up....but i dont know, why my easy query select count (*) from cdr use 90% of Yarn ..?

take a look at screenshots..

[#8 · Accepted solution]

@enzo EL

Please also try to setup the following parameters:

set hive.tez.java.opts=-Xmx1640m (0.8 times hive.tez.container.size)
set tez.runtime.io.sort.mb=820 (0.4 times hive.tez.container.size)
set tez.runtime.unordered.output.buffer.size-mb=205 (0.1 times hive.tez.container.size)

The following link provides good examples and explaination on Tez Tuning.
https://community.hortonworks.com/articles/14309/demystify-tez-tuning-step-by-step.html

.

[#9]

Thank you....

According to your posted links and this link https://community.hortonworks.com/questions/67677/javalangoutofmemoryerror-java-heap-space-error.htm... I solved the problem.

Thank you.

[#10]

@enzo EL

Good to know that the issue is resolved. As the issue is resolved, hence it will be also great if you can mark this HCC thread as Answered by clicking on the "Accept" Button on the correct answer. That way other HCC users can quickly find the solution when they encounter the same issue.