Member since
07-16-2019
1
Post
0
Kudos Received
0
Solutions
07-16-2019
03:23 PM
Hi, We have 4 node HDP cluster installed, 3 data nodes and 1 namenode. I have enabled Hive LLAP for low latency queries but its behavior is strange, for example when I try to execute a simple count query it works fine but the same query fails on executing again with following error message. Error Message ERROR : Status: Failed
ERROR : Vertex re-running, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00
ERROR : Vertex re-running, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00
ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00, diagnostics=[Vertex vertex_1563258949712_0034_51_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE, Vertex vertex_1563258949712_0034_51_00 [Map 1] failed as task task_1563258949712_0034_51_00_000004 failed after vertex succeeded.]
ERROR : DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0 Detailed Error Message ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00Vertex re-running, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00Vertex failed, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00, diagnostics=[Vertex vertex_1563258949712_0034_51_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE, Vertex vertex_1563258949712_0034_51_00 [Map 1] failed as task task_1563258949712_0034_51_00_000004 failed after vertex succeeded.]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
INFO : Completed executing command(queryId=hive_20190716163641_303daa3b-62b4-499a-9d3a-ab644e77c739); Time taken: 26.412 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00Vertex re-running, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00Vertex failed, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00, diagnostics=[Vertex vertex_1563258949712_0034_51_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE, Vertex vertex_1563258949712_0034_51_00 [Map 1] failed as task task_1563258949712_0034_51_00_000004 failed after vertex succeeded.]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0 (state=08S01,code=2) Total cluster memory is 384 GB with 56 cores and LLAP queue is allocated with 45% resources in yarn queue manager. I have tweaked many setting after reading countless articles but none of them are helping me to resolve the issue. A little help or hint towards right direction would be great.
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Tez