Support Questions

Find answers, ask questions, and share your expertise

Why is Hive running out of memory OOM on an Azure VM with 4 cores and 28Gb of Ram (Sandbox)

avatar
Contributor

Hi, I am trying to run some simple enough queries on my files (250000 records) and I am receiving an out of memory message as follows:

ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1458912652919_0002_1_00, diagnostics=[Task failed, taskId=task_1458912652919_0002_1_00_000006, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap spaceat org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:159)at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139).........and this continues for many lines. 

I have tried researching this issue and I came across this post but I am lost as to how I can modify the settings discussed: https://azure.microsoft.com/en-us/blog/hive-memory-settings-resolve-out-of-memory-errors-using-azure...

Can anyone shed any light on this for me please? By the way, below is the query I was trying to run on my Hive table:

SELECT * FROM pigoutputhive SORT BY Score DESC LIMIT 10;

Many thanks in advance.

1 ACCEPTED SOLUTION

avatar
Contributor

So I finally figured out my problem, it had to do with my own impatience!

Although Hive said the query ran successfully and had not returned any results, it was still working away in the background and eventually spat back some results. Although the results were not correct that's for another day.

I changed the settings as per the linked article above but needed an Admin profile to do so and it eventually worked.

View solution in original post

4 REPLIES 4

avatar
Contributor

I came across this article and changed the recommended settings but when I now run the above query I receive no errors or results even though the status is completed. Has anyone any other suggestions please?

avatar
Contributor

So I finally figured out my problem, it had to do with my own impatience!

Although Hive said the query ran successfully and had not returned any results, it was still working away in the background and eventually spat back some results. Although the results were not correct that's for another day.

I changed the settings as per the linked article above but needed an Admin profile to do so and it eventually worked.

avatar

@John Garrigan thank you for posting your resolution. Much appreciated!

avatar
Explorer

how do you login as admin profile? I am using the Maria_dev, it is all grey out in config, I could not change anything