Hive error: Vertex failed

[#1]

When I try to insert data from a table into a partitioned bucketed table, I am getting this error:

Status: Failed Vertex failed, vertexName=Map 1, vertexId=vertex_1490155524314_0037_1_00, diagnostics=[Task failed, taskId=task_1490155524314_0037_1_00_000007, diagnostics=[TaskAttempt 0 failed, info=[attempt_1490155524314_0037_1_00_000007_0 being failed for too many output errors. failureFraction=0.125, MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, readErrorTimespan=0], TaskAttempt 1 failed, info=[attempt_1490155524314_0037_1_00_000007_1 being failed for too many output errors. failureFraction=0.125, MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, readErrorTimespan=0], TaskAttempt 2 failed, info=[attempt_1490155524314_0037_1_00_000007_2 being failed for too many output errors. failureFraction=0.125, MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, readErrorTimespan=0], TaskAttempt 3 failed, info=[attempt_1490155524314_0037_1_00_000007_3 being failed for too many output errors. failureFraction=0.125, MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, readErrorTimespan=0]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:14, Vertex vertex_1490155524314_0037_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE] Vertex killed, vertexName=Reducer 2, vertexId=vertex_1490155524314_0037_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:8, Vertex vertex_1490155524314_0037_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE] DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1 FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1490155524314_0037_1_00, diagnostics=[Task failed, taskId=task_1490155524314_0037_1_00_000007, diagnostics=[TaskAttempt 0 failed, info=[attempt_1490155524314_0037_1_00_000007_0 being failed for too many output errors. failureFraction=0.125, MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, readErrorTimespan=0], TaskAttempt 1 failed, info=[attempt_1490155524314_0037_1_00_000007_1 being failed for too many output errors. failureFraction=0.125, MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, readErrorTimespan=0], TaskAttempt 2 failed, info=[attempt_1490155524314_0037_1_00_000007_2 being failed for too many output errors. failureFraction=0.125, MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, readErrorTimespan=0], TaskAttempt 3 failed, info=[attempt_1490155524314_0037_1_00_000007_3 being failed for too many output errors. failureFraction=0.125, MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, readErrorTimespan=0]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:14, Vertex vertex_1490155524314_0037_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1490155524314_0037_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:8, Vertex vertex_1490155524314_0037_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1

[#2]

Can you see if this helps:

https://community.hortonworks.com/questions/72569/vertex-failed-while-overwrite-the-hive-table.html

[#3]

Thanks for the reply Namit. It didn't help. I think it has to do with configuring my memory parameters.

[#4]

Will need to see the output from YARN application logs, you can fetch them by running

yarn logs -applicaitonId application_1490155524314_0037

[#5]

It's a Tez issue. Update the below parameters in tez-site and hive-site. It might resolve your issue.

Tez-site.xml:

<name>tez.am.launch.cmd-opts</name>

<value>-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC</value

<name>tez.task.launch.cmd-opts</name>

<value>-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC</value>

Hive-site.xm:

<name>hive.tez.java.opts</name>

<value>-server -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseParallelGC -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps</value>

[#6]

The GC algorithm is G1GC. Could that be the issue?

[#7]

Try reducing the tez.runtime.shuffle.fetch.buffer.percent. More information here:

https://community.hortonworks.com/content/supportkb/49030/hive-query-in-tez-reducer-fails-with-javal...

[#8]

Thanks for working with me on this Deepesh. The issue was with the user not having permissions.

[#9]

Can you elaborate on the missing permissions? I am running into a similar issue. Thanks.

[#10]

Kerra, do you mind sharing your solution with your user permissions?

I have this query: SELECT ingest_dt, COUNT(*) FROM table GROUP BY ingest_dt

Some users run it successfully but others not, and I am not able to find the problem.

Thanks.