Support Questions
Find answers, ask questions, and share your expertise

Hive query: DAG did not succeed due to VERTEX_FAILURE

Hive query: DAG did not succeed due to VERTEX_FAILURE

Hi

I am using HDP-2.6.3, kerberos enabled and synced with AD. Ranger plugin is enabled for hive.

I am trying to run the below query:

set hive.support.sql11.reserved.keywords=false;

select timestamp_, cid, type_, date from test_data where date = '2018-03-13' order by date desc;

When i run this query from hive shell, the query works fine.

But when i run this query from Zeppelin via an AD user, the query failed with error mentioned below. If i run the same query from Hive view using the same AD user, it fails there as well. Also, it doesn't fails all the time, it has worked couple of times for the same user, but most of times it fails.

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica}

hive> SHOW CREATE TABLE test_data;

OK

CREATE EXTERNAL TABLE `test_data`(

`timestamp_` timestamp,

`event` string,

`type_` string)

PARTITIONED BY (

`cid` string,

`date` date)

ROW FORMAT SERDE

'org.apache.hadoop.hive.ql.io.orc.OrcSerde'

STORED AS INPUTFORMAT

'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'

OUTPUTFORMAT

'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'

LOCATION

'hdfs://testcluster/user/admsas/test_events/data'

TBLPROPERTIES (

'transient_lastDdlTime'='1521037297')

java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map 1, vertexId=vertex_1522659663092_0004_2_00Vertex failed, vertexName=Map 1, vertexId=vertex_1522659663092_0004_2_00, diagnostics=[Task failed, taskId=task_1522659663092_0004_2_00_000002, diagnostics=[TaskAttempt 0 failed, info=[attempt_1522659663092_0004_2_00_000002_0 being failed for too many output errors. failureFraction=1.0, MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, readErrorTimespan=0], TaskAttempt 1 failed, info=[attempt_1522659663092_0004_2_00_000002_1 being failed for too many output errors. failureFraction=1.0, MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, readErrorTimespan=0], TaskAttempt 2 failed, info=[attempt_1522659663092_0004_2_00_000002_2 being failed for too many output errors. failureFraction=1.0, MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, readErrorTimespan=0], TaskAttempt 3 failed, info=[attempt_1522659663092_0004_2_00_000002_3 being failed for too many output errors. failureFraction=1.0, MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, readErrorTimespan=0]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:1, Vertex vertex_1522659663092_0004_2_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1522659663092_0004_2_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1, Vertex vertex_1522659663092_0004_2_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1 at org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:354)

Not able to find the root cause.

Your help would be appreciated.

Thanks