Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Why is Hive running out of memory OOM on an Azure VM with 4 cores and 28Gb of Ram (Sandbox)

SOLVED Go to solution
Highlighted

Why is Hive running out of memory OOM on an Azure VM with 4 cores and 28Gb of Ram (Sandbox)

New Contributor

Hi, I am trying to run some simple enough queries on my files (250000 records) and I am receiving an out of memory message as follows:

ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1458912652919_0002_1_00, diagnostics=[Task failed, taskId=task_1458912652919_0002_1_00_000006, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap spaceat org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:159)at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139).........and this continues for many lines. 

I have tried researching this issue and I came across this post but I am lost as to how I can modify the settings discussed: https://azure.microsoft.com/en-us/blog/hive-memory-settings-resolve-out-of-memory-errors-using-azure...

Can anyone shed any light on this for me please? By the way, below is the query I was trying to run on my Hive table:

SELECT * FROM pigoutputhive SORT BY Score DESC LIMIT 10;

Many thanks in advance.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Why is Hive running out of memory OOM on an Azure VM with 4 cores and 28Gb of Ram (Sandbox)

New Contributor

So I finally figured out my problem, it had to do with my own impatience!

Although Hive said the query ran successfully and had not returned any results, it was still working away in the background and eventually spat back some results. Although the results were not correct that's for another day.

I changed the settings as per the linked article above but needed an Admin profile to do so and it eventually worked.

4 REPLIES 4

Re: Why is Hive running out of memory OOM on an Azure VM with 4 cores and 28Gb of Ram (Sandbox)

New Contributor

I came across this article and changed the recommended settings but when I now run the above query I receive no errors or results even though the status is completed. Has anyone any other suggestions please?

Re: Why is Hive running out of memory OOM on an Azure VM with 4 cores and 28Gb of Ram (Sandbox)

New Contributor

So I finally figured out my problem, it had to do with my own impatience!

Although Hive said the query ran successfully and had not returned any results, it was still working away in the background and eventually spat back some results. Although the results were not correct that's for another day.

I changed the settings as per the linked article above but needed an Admin profile to do so and it eventually worked.

Re: Why is Hive running out of memory OOM on an Azure VM with 4 cores and 28Gb of Ram (Sandbox)

@John Garrigan thank you for posting your resolution. Much appreciated!

Re: Why is Hive running out of memory OOM on an Azure VM with 4 cores and 28Gb of Ram (Sandbox)

New Contributor

how do you login as admin profile? I am using the Maria_dev, it is all grey out in config, I could not change anything