Support Questions

Find answers, ask questions, and share your expertise

Vertex did not succeed due to OWN_TASK_FAILURE

avatar
Contributor

Hi,

when I issue the following insert command 'insert into table test values(1,"name")', I am getting the error:

jdbc:hive2://host.domain.local> insert into table test values(1,"name");
INFO :Compiling command(queryId=hive_20181108185449_e3ac40a8-b413-442e-8cd3-3a5aaeb0da30): insert into table test values(1,"name")
INFO :SemanticAnalysisCompleted(retrial =false)
INFO :ReturningHive schema:Schema(fieldSchemas:[FieldSchema(name:col1, type:int, comment:null),FieldSchema(name:col2, type:string, comment:null)], properties:null)
INFO :Completed compiling command(queryId=hive_20181108185449_e3ac40a8-b413-442e-8cd3-3a5aaeb0da30);Time taken:0.245 seconds
INFO :Executing command(queryId=hive_20181108185449_e3ac40a8-b413-442e-8cd3-3a5aaeb0da30): insert into table test values(1,"name")
INFO :Query ID = hive_20181108185449_e3ac40a8-b413-442e-8cd3-3a5aaeb0da30
INFO :Total jobs =1
INFO :LaunchingJob1out of 1
INFO :Starting task [Stage-1:MAPRED]in serial mode
INFO :Subscribed to counters:[]for queryId: hive_20181108185449_e3ac40a8-b413-442e-8cd3-3a5aaeb0da30
INFO :Tez session hasn't been created yet. Opening session
INFO : Dag name: insert into table test values(1,"name") (Stage-1)
INFO : Status: Running (Executing on YARN cluster with App id application_1541680241474_0006)
----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
----------------------------------------------------------------------------------------------
Map 1 container RUNNING 1 0 0 1 4 0
Reducer 2 container INITED 1 0 0 1 0 0
----------------------------------------------------------------------------------------------
VERTICES: 00/02 [>>--------------------------] 0% ELAPSED TIME: 7.50 s
----------------------------------------------------------------------------------------------
ERROR : Status: Failed
ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1541680241474_0006_1_00, diagnostics=[Task failed, taskId=task_1541680241474_0006_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1541680241474_0006_1_00_000000_0:java.lang.IllegalArgumentException
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
at org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:177)
...
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) : attempt_1541680241474_0006_1_00_000000_1:java.lang.IllegalArgumentException
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
at org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:177)
...
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 2 failed, info=[Error: Error while running task ( failure ) : attempt_1541680241474_0006_1_00_000000_2:java.lang.IllegalArgumentException
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
at org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:177)
...
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 3 failed, info=[Error: Error while running task ( failure ) : attempt_1541680241474_0006_1_00_000000_3:java.lang.IllegalArgumentException
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
at org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:177)
...
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1541680241474_0006_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]
ERROR : Vertex killed, vertexName=Reducer 2, vertexId=vertex_1541680241474_0006_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1, Vertex vertex_1541680241474_0006_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]
ERROR : DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1
INFO : org.apache.tez.common.counters.DAGCounter:
INFO : NUM_FAILED_TASKS: 4
INFO : TOTAL_LAUNCHED_TASKS: 4
INFO : AM_CPU_MILLISECONDS: 2460
INFO : AM_GC_TIME_MILLIS: 0
ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1541680241474_0006_1_00, diagnostics=[Task failed, taskId=task_1541680241474_0006_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1541680241474_0006_1_00_000000_0:java.lang.IllegalArgumentException
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
at org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:177)
...
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) : attempt_1541680241474_0006_1_00_000000_1:java.lang.IllegalArgumentException
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
at org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:177)
...
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 2 failed, info=[Error: Error while running task ( failure ) : attempt_1541680241474_0006_1_00_000000_2:java.lang.IllegalArgumentException
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
at org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:177)
...
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 3 failed, info=[Error: Error while running task ( failure ) : attempt_1541680241474_0006_1_00_000000_3:java.lang.IllegalArgumentException
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
at org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:177)
...
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1541680241474_0006_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1541680241474_0006_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1, Vertex vertex_1541680241474_0006_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1
INFO : Completed executing command(queryId=hive_20181108185449_e3ac40a8-b413-442e-8cd3-3a5aaeb0da30); Time taken: 12.202 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1541680241474_0006_1_00, diagnostics=[Task failed, taskId=task_1541680241474_0006_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1541680241474_0006_1_00_000000_0:java.lang.IllegalArgumentException
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
at org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:177)
...
at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) : attempt_1541680241474_0006_1_00_000000_1:java.lang.IllegalArgumentException
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
...
at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 2 failed, info=[Error: Error while running task ( failure ) : attempt_1541680241474_0006_1_00_000000_2:java.lang.IllegalArgumentException
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
at org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:177)
...
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
...
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 3 failed, info=[Error: Error while running task ( failure ) : attempt_1541680241474_0006_1_00_000000_3:java.lang.IllegalArgumentException
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
at org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:177)
...
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
...
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1541680241474_0006_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1541680241474_0006_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1, Vertex vertex_1541680241474_0006_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1 (state=08S01,code=2)
0: jdbc:hive2://host.domain.local>

I have tried changing the container size in YARN, Hive, Tez but no success. I also set the recommendation values provided by yarn-utils.py but it also is not helped.

2 REPLIES 2

avatar
Explorer

same problem here.... HDP 3.0.1

avatar
Super Collaborator

Try the following, lets assume your hive.tez.container.size=2048.

set hive.tez.java.opts=-Xmx1640m (0.8 times hive.tez.container.size)

set tez.runtime.io.sort.mb=820 (0.4 times hive.tez.container.size)

set tez.runtime.unordered.output.buffer.size-mb=205 (0.1 times hive.tez.container.size)