Skip to main content
drgenious
Silver ⭐️⭐️⭐️
November 23, 2022
Question

Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask.

I have a job which runs a hive query inside. When it comes the time for the query Oozie throws this error:

 

Error while compiling statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map 1, vertexId=vertex_1668428709182_0049_1_00Vertex re-running, vertexName=Map 1, vertexId=vertex_1668428709182_0049_1_00Vertex re-running, vertexName=Map 1, vertexId=vertex_1668428709182_0049_1_00Vertex failed, vertexName=Map 1, vertexId=vertex_1668428709182_0049_1_00, diagnostics=[Vertex vertex_1668428709182_0049_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE, Vertex vertex_1668428709182_0049_1_00 [Map 1] failed as task task_1668428709182_0049_1_00_000000 failed after vertex succeeded.]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0

 

I can not understand a lot of this error but when I run the job through terminal it ends successfully.

6 replies

Shahrukh_shaikh
Silver ⭐️⭐️
November 23, 2022

Can you please share the app logs? it looks like data issue.

Translate
drgenious
drgeniousAuthor
Silver ⭐️⭐️⭐️
November 23, 2022

Hi @Shahrukh_shaikh. I do not have them now. What do you mean data issue? When I run theough terminal everything runs smoothly

 

Translate
Shahrukh_shaikh
Silver ⭐️⭐️
November 23, 2022

most of the time vertex error comes when you have something wrong with data. Check if any data offloading or transforming is happening.

Translate
asish
Gold ⭐️⭐️⭐️
Gold ⭐️⭐️⭐️
November 24, 2022

@drgenious Can you please provide below:

 

 1. oozie wf logs(oozie application logs) ==> i can find the exact hive query executed.

 

2. Hive application logs.

 

Please collect it using:

 

    yarn logs -applicationId <appid> > <appid>.txt ==> FOr bith Oozie and hive

Translate
Rami_Sunkara
Silver ⭐️⭐️
February 20, 2024

@Shahrukh_shaikh - We are getting the below error when we are running the select distinct on column where it has all NULL values.  Does that mean that we should not be running distinct on such columns. 

[Cloudera][Hardy] (35) Error from server: error code: '2' error message: 'Error while compiling statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1708122343209_1093_1_00, diagnostics=[Task failed, taskId=task_1708122343209_1093_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1708122343209_1093_1_00_000000_0:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:298)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:252)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:62)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:38)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:101)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:437)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:269)
... 16 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row
at org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:970)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
... 19 more
Caused by: java.lang.RuntimeException: no value set

Translate
Rami_Sunkara
Silver ⭐️⭐️
February 21, 2024

Got to know from the dev team that they have modified the column definition. We ran the MSCK repair table and we are able to run the select distinct query. 

Vertex errors may not relate to memory issues. Hope this helps the community. 

Translate