Community Articles

Find and share helpful community-sourced technical articles.
Celebrating as our community reaches 100,000 members! Thank you!
Labels (1)
Rising Star

ISSUE: Java Heap Space issue in Hive MR engine

While working on a sample data set in hive. Query such as "select count(*)" was seen to fail with below error.

Starting Job = job_1491603076412_0001, Tracking URL = http://krishna3.openstacklocal:8088/proxy/application_1491603076412_0001/
Kill Command = /usr/hdp/ job  -kill job_1491603076412_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2017-04-07 22:18:09,736 Stage-1 map = 0%,  reduce = 0%
2017-04-07 22:18:46,065 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_1491603076412_0001 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1491603076412_0001_m_000000 (and more) from job job_1491603076412_0001

Task with the most failures(4):
Task ID:

Diagnostic Messages for this Task:
Error: Java heap space

FAILED: Execution Error, return code 2 from
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

Checking the corresponding application logs, we observe that

2017-04-07 22:25:40,828 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(
	at org.apache.hadoop.mapred.MapTask.createSortingCollector(
	at org.apache.hadoop.mapred.MapTask.runOldMapper(
	at org.apache.hadoop.mapred.YarnChild$
	at Method)
	at org.apache.hadoop.mapred.YarnChild.main(

ROOT CAUSE: Insufficient HeapSpace in MR Engine for


Increasing from 1.2G to 1.75G and hence increasing the to 1003 to -Xmx1433m

and restarting the necessary services did resolve the problem.

(NOTE: and value recommendations were made by Ambari )