Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Tez out of memory?

Solved Go to solution

Tez out of memory?

Hi guys,

I am trying to execute a command from zeppelin. I use the sandbox 2.6 and my VM has 16GB ram.

I dont know why it is out of memmory. Could anybody help me?

43556-screenshot-48.png

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Tez out of memory?

Super Mentor

@enzo EL

Please also try to setup the following parameters:

set hive.tez.java.opts=-Xmx1640m (0.8 times hive.tez.container.size)
set tez.runtime.io.sort.mb=820 (0.4 times hive.tez.container.size)
set tez.runtime.unordered.output.buffer.size-mb=205 (0.1 times hive.tez.container.size)


The following link provides good examples and explaination on Tez Tuning.
https://community.hortonworks.com/articles/14309/demystify-tez-tuning-step-by-step.html

.

View solution in original post

9 REPLIES 9
Highlighted

Re: Tez out of memory?

Super Mentor

@enzo EL

Where do you see the OutOfMemory ? Can you please share the complete trace of OutOfMemory and where do you see it.

What is your "hive.tez.container.size" value ? (Can you try increasing it and then see if it helps).


If the Hive Table is ORC formatted then you can also refer to the following article for tuning: https://community.hortonworks.com/articles/68631/optimizing-hive-queries-for-orc-formatted-tables.ht...

.

Highlighted

Re: Tez out of memory?

43558-screenshot-49.png

oh sorry...this will be better. Where could I foung info abou "hive.tez.container.size" value ?

Thanks

Highlighted

Re: Tez out of memory?

Super Mentor

@enzo EL

Ambari UI --> Hive --> Configs --> Settings Tab

hive.tez.container.size

42543-tez-container-size.png

.

Highlighted

Re: Tez out of memory?

No I set the value to 900MB and it is still not working...

Highlighted

Re: Tez out of memory?

Super Mentor

@enzo EL

Thats a tuning parameter .. so it's value depends on your requirement or the kind of job that you are running.

Please try to set it to a higher value like 2500MB and then see if it works.

Highlighted

Re: Tez out of memory?

I set it up....but i dont know, why my easy query select count (*) from cdr use 90% of Yarn ..?

take a look at screenshots..

43560-screenshot-52.png

43561-screenshot-53.png

Highlighted

Re: Tez out of memory?

Super Mentor

@enzo EL

Please also try to setup the following parameters:

set hive.tez.java.opts=-Xmx1640m (0.8 times hive.tez.container.size)
set tez.runtime.io.sort.mb=820 (0.4 times hive.tez.container.size)
set tez.runtime.unordered.output.buffer.size-mb=205 (0.1 times hive.tez.container.size)


The following link provides good examples and explaination on Tez Tuning.
https://community.hortonworks.com/articles/14309/demystify-tez-tuning-step-by-step.html

.

View solution in original post

Highlighted

Re: Tez out of memory?

Thank you....

According to your posted links and this link https://community.hortonworks.com/questions/67677/javalangoutofmemoryerror-java-heap-space-error.htm... I solved the problem.

Thank you.

Highlighted

Re: Tez out of memory?

Super Mentor

@enzo EL

Good to know that the issue is resolved. As the issue is resolved, hence it will be also great if you can mark this HCC thread as Answered by clicking on the "Accept" Button on the correct answer. That way other HCC users can quickly find the solution when they encounter the same issue.

Don't have an account?
Coming from Hortonworks? Activate your account here