Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Tez out of memory?

avatar

Hi guys,

I am trying to execute a command from zeppelin. I use the sandbox 2.6 and my VM has 16GB ram.

I dont know why it is out of memmory. Could anybody help me?

43556-screenshot-48.png

1 ACCEPTED SOLUTION

avatar
Master Mentor

@enzo EL

Please also try to setup the following parameters:

set hive.tez.java.opts=-Xmx1640m (0.8 times hive.tez.container.size)
set tez.runtime.io.sort.mb=820 (0.4 times hive.tez.container.size)
set tez.runtime.unordered.output.buffer.size-mb=205 (0.1 times hive.tez.container.size)


The following link provides good examples and explaination on Tez Tuning.
https://community.hortonworks.com/articles/14309/demystify-tez-tuning-step-by-step.html

.

View solution in original post

9 REPLIES 9

avatar
Master Mentor

@enzo EL

Where do you see the OutOfMemory ? Can you please share the complete trace of OutOfMemory and where do you see it.

What is your "hive.tez.container.size" value ? (Can you try increasing it and then see if it helps).


If the Hive Table is ORC formatted then you can also refer to the following article for tuning: https://community.hortonworks.com/articles/68631/optimizing-hive-queries-for-orc-formatted-tables.ht...

.

avatar

43558-screenshot-49.png

oh sorry...this will be better. Where could I foung info abou "hive.tez.container.size" value ?

Thanks

avatar
Master Mentor

@enzo EL

Ambari UI --> Hive --> Configs --> Settings Tab

hive.tez.container.size

42543-tez-container-size.png

.

avatar

No I set the value to 900MB and it is still not working...

avatar
Master Mentor

@enzo EL

Thats a tuning parameter .. so it's value depends on your requirement or the kind of job that you are running.

Please try to set it to a higher value like 2500MB and then see if it works.

avatar

I set it up....but i dont know, why my easy query select count (*) from cdr use 90% of Yarn ..?

take a look at screenshots..

43560-screenshot-52.png

43561-screenshot-53.png

avatar
Master Mentor

@enzo EL

Please also try to setup the following parameters:

set hive.tez.java.opts=-Xmx1640m (0.8 times hive.tez.container.size)
set tez.runtime.io.sort.mb=820 (0.4 times hive.tez.container.size)
set tez.runtime.unordered.output.buffer.size-mb=205 (0.1 times hive.tez.container.size)


The following link provides good examples and explaination on Tez Tuning.
https://community.hortonworks.com/articles/14309/demystify-tez-tuning-step-by-step.html

.

avatar

Thank you....

According to your posted links and this link https://community.hortonworks.com/questions/67677/javalangoutofmemoryerror-java-heap-space-error.htm... I solved the problem.

Thank you.

avatar
Master Mentor

@enzo EL

Good to know that the issue is resolved. As the issue is resolved, hence it will be also great if you can mark this HCC thread as Answered by clicking on the "Accept" Button on the correct answer. That way other HCC users can quickly find the solution when they encounter the same issue.