Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask.

[#1]

I have a job which runs a hive query inside. When it comes the time for the query Oozie throws this error:

Error while compiling statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map 1, vertexId=vertex_1668428709182_0049_1_00Vertex re-running, vertexName=Map 1, vertexId=vertex_1668428709182_0049_1_00Vertex re-running, vertexName=Map 1, vertexId=vertex_1668428709182_0049_1_00Vertex failed, vertexName=Map 1, vertexId=vertex_1668428709182_0049_1_00, diagnostics=[Vertex vertex_1668428709182_0049_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE, Vertex vertex_1668428709182_0049_1_00 [Map 1] failed as task task_1668428709182_0049_1_00_000000 failed after vertex succeeded.]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0

I can not understand a lot of this error but when I run the job through terminal it ends successfully.

[#2]

Can you please share the app logs? it looks like data issue.

[#3]

Hi @Shahrukh_shaikh. I do not have them now. What do you mean data issue? When I run theough terminal everything runs smoothly

[#4]

most of the time vertex error comes when you have something wrong with data. Check if any data offloading or transforming is happening.

[#5]

@drgenious Can you please provide below:

 1. oozie wf logs(oozie application logs) ==> i can find the exact hive query executed.

2. Hive application logs.

Please collect it using:

    yarn logs -applicationId <appid> > <appid>.txt ==> FOr bith Oozie and hive

[#6]

@Shahrukh_shaikh - We are getting the below error when we are running the select distinct on column where it has all NULL values.  Does that mean that we should not be running distinct on such columns. 

[Cloudera][Hardy] (35) Error from server: error code: '2' error message: 'Error while compiling statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1708122343209_1093_1_00, diagnostics=[Task failed, taskId=task_1708122343209_1093_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1708122343209_1093_1_00_000000_0:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:298)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:252)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:62)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:38)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:101)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:437)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:269)
... 16 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row
at org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:970)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
... 19 more
Caused by: java.lang.RuntimeException: no value set

[#7]

Got to know from the dev team that they have modified the column definition. We ran the MSCK repair table and we are able to run the select distinct query. 

Vertex errors may not relate to memory issues. Hope this helps the community.