Support Questions
Find answers, ask questions, and share your expertise

Hive LLAP Vertex failure

Hive LLAP Vertex failure

New Contributor

Hi,

We have 4 node HDP cluster installed, 3 data nodes and 1 namenode.

I have enabled Hive LLAP for low latency queries but its behavior is strange, for example when I try to execute a simple count query it works fine but the same query fails on executing again with following error message.


Error Message

ERROR : Status: Failed
ERROR : Vertex re-running, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00
ERROR : Vertex re-running, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00
ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00, diagnostics=[Vertex vertex_1563258949712_0034_51_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE, Vertex vertex_1563258949712_0034_51_00 [Map 1] failed as task task_1563258949712_0034_51_00_000004 failed after vertex succeeded.]
ERROR : DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0


Detailed Error Message

ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00Vertex re-running, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00Vertex failed, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00, diagnostics=[Vertex vertex_1563258949712_0034_51_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE, Vertex vertex_1563258949712_0034_51_00 [Map 1] failed as task task_1563258949712_0034_51_00_000004 failed after vertex succeeded.]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
INFO  : Completed executing command(queryId=hive_20190716163641_303daa3b-62b4-499a-9d3a-ab644e77c739); Time taken: 26.412 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00Vertex re-running, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00Vertex failed, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00, diagnostics=[Vertex vertex_1563258949712_0034_51_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE, Vertex vertex_1563258949712_0034_51_00 [Map 1] failed as task task_1563258949712_0034_51_00_000004 failed after vertex succeeded.]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0 (state=08S01,code=2)

Total cluster memory is 384 GB with 56 cores and LLAP queue is allocated with 45% resources in yarn queue manager.

I have tweaked many setting after reading countless articles but none of them are helping me to resolve the issue.

A little help or hint towards right direction would be great.


1 REPLY 1

Re: Hive LLAP Vertex failure

Contributor

@ateeq ashraf,


Are you using a cache, can you check if the cache is full?