Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Failure to execute Hive Query from Lab 2

Failure to execute Hive Query from Lab 2

New Contributor

I am using Azure to launch my HDP Sandbox..

I have followed all the steps in Lab 0 and Lab 1. I even retried all the steps leading to the section "2.6.2 Load Sample Data of truck_milage"

When I execute the query:

SELECT truckid, avg(mpg) avgmpg FROM truck_mileage GROUP BY truckid;

It does not succeed and returns this error:

NFO  : Session is already open
INFO  : Dag name: CREATE TABLE avg_mileage AS select...truckid(Stage-1)
INFO  : INFO  : Status: Running (Executing on YARN cluster with App id application_1463676341868_0006)INFO  : Map 1: -/-	Reducer 2: 0/1	
INFO  : Map 1: 0/1	Reducer 2: 0/1	
INFO  : Map 1: 0/1	Reducer 2: 0/1	
INFO  : Map 1: 0(+1)/1	Reducer 2: 0/1	
INFO  : Map 1: 0(+1)/1	Reducer 2: 0/1	
INFO  : Map 1: 0(+1,-1)/1	Reducer 2: 0/1	
INFO  : Map 1: 0(+1,-1)/1	Reducer 2: 0/1	
INFO  : Map 1: 0(+1,-1)/1	Reducer 2: 0/1	
INFO  : Map 1: 0(+1,-2)/1	Reducer 2: 0/1	
INFO  : Map 1: 0(+1,-2)/1	Reducer 2: 0/1	
INFO  : Map 1: 0(+1,-2)/1	Reducer 2: 0/1	
INFO  : Map 1: 0(+1,-3)/1	Reducer 2: 0/1	
INFO  : Map 1: 0(+1,-3)/1	Reducer 2: 0/1	
INFO  : Map 1: 0(+1,-3)/1	Reducer 2: 0/1	
ERROR : Status: Failed
ERROR
 : Vertex failed, vertexName=Map 1, 
vertexId=vertex_1463676341868_0006_2_00, diagnostics=[Task failed, 
taskId=task_1463676341868_0006_2_00_000000, diagnostics=[TaskAttempt 0 
failed, info=[Error: Failure while running 
task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap 
space
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:159)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
	at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
	at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:172)
	at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:116)
	at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:142)
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:142)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:149)
	... 14 more
],
 TaskAttempt 1 failed, info=[Error: Failure while running 
task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap 
space
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:159)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
	at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
	at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:172)
	at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:116)
	at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:142)
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:142)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:149)
	... 14 more
],
 TaskAttempt 2 failed, info=[Error: Failure while running 
task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap 
space
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:159)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
	at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
	at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:172)
	at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:116)
	at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:142)
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:142)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:149)
	... 14 more
],
 TaskAttempt 3 failed, info=[Error: Failure while running 
task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap 
space
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:159)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
	at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
	at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:172)
	at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:116)
	at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:142)
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:142)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:149)
	... 14 more
]],
 Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 
killedTasks:0, Vertex vertex_1463676341868_0006_2_00 [Map 1] 
killed/failed due to:OWN_TASK_FAILURE]
ERROR : Vertex killed, 
vertexName=Reducer 2, vertexId=vertex_1463676341868_0006_2_01, 
diagnostics=[Vertex received Kill while in RUNNING state., Vertex did 
not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1, 
Vertex vertex_1463676341868_0006_2_01 [Reducer 2] killed/failed due 
to:OTHER_VERTEX_FAILURE]
ERROR : DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1
8 REPLIES 8

Re: Failure to execute Hive Query from Lab 2

New Contributor

Any help would be greatly appreciated: I am completely new to Hadoop and Hortonworks

Re: Failure to execute Hive Query from Lab 2

New Contributor

I experience exactly the same problem while working through Lab 2.

Re: Failure to execute Hive Query from Lab 2

You are getting the out of memory exception Try doing the following and retry.

for /etc/tez/conf/hive-site.xml. Try increasing the values for following (set Xmx and Xms same).

  • hive.tez.container.size:
  • hive.tez.java.opts: -> -Xmx -Xms

Re: Failure to execute Hive Query from Lab 2

Super Collaborator

@Nam Thai

Adding on to the answer what @milind pandit has already provided.

As a workaround, we can adjust hive-tez parameters from Ambari. Go to the Hive page, click on Configs tab and make the changes in two properties:

  • hive.tez.container.size: from 200 to 682 MB (as recommended by Ambari)
  • hive.tez.java.opts: -> from -server -Xmx200m -Djava.net.preferIPv4Stack=true to -server -Xmx512m -Djava.net.preferIPv4Stack=true.

Re: Failure to execute Hive Query from Lab 2

New Contributor

Tried many solutions , not working . Time being using this work around -

CREATE TABLE avg_mileage (truckid STRING,avgmpg BIGINT ) STORED AS ORC

Re: Failure to execute Hive Query from Lab 2

New Contributor

I think this is defect in the latest sandbox. Ideally a new learner should be able to perform basic assignments without having admin knowledge like modifying/determining optimum container size, heap size or other parameters etc.

Re: Failure to execute Hive Query from Lab 2

Rising Star

Hi @Laxmikant Deshmukh we agree. We are looking into this, since this issue is not present in our on prem sandbox with the same settings.

Regards,

Robert

Re: Failure to execute Hive Query from Lab 2

New Contributor

Hi. I have the same error.

I work with virtual box image:

sandbox-version command output--> Sandbox information: Created on: 14_03_2016_15_03_14 for virtualbox Hadoop stack version: Hadoop 2.7.1.2.4.0.0-169 Ambari Version: 2.2.1.0 Ambari Hash: 6a8abfa65789b87da764549c27ca0f1440b91297 Ambari build: Release : 161 Java version: 1.7.0_95 OS Version: CentOS release 6.7 (Final)

I increment the memory of my latop from 8 to 16 GB and apply the configuration changes from mrizvi answer, and this solve this error, but thask not end ever.

Finally change the hive.execution.engine from tez to mr and this work for me (on Ambari Hive configuration panel change Optimization Tez Execution Engine to MapReduce value).

I´m understand that this is a undesirable workaround but...

Regards.