Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive LLAP Vertex failure

Highlighted

Hive LLAP Vertex failure

New Contributor

Hi,

We have 4 node HDP cluster installed, 3 data nodes and 1 namenode.

I have enabled Hive LLAP for low latency queries but its behavior is strange, for example when I try to execute a simple count query it works fine but the same query fails on executing again with following error message.


Error Message

ERROR : Status: Failed
ERROR : Vertex re-running, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00
ERROR : Vertex re-running, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00
ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00, diagnostics=[Vertex vertex_1563258949712_0034_51_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE, Vertex vertex_1563258949712_0034_51_00 [Map 1] failed as task task_1563258949712_0034_51_00_000004 failed after vertex succeeded.]
ERROR : DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0


Detailed Error Message

ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00Vertex re-running, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00Vertex failed, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00, diagnostics=[Vertex vertex_1563258949712_0034_51_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE, Vertex vertex_1563258949712_0034_51_00 [Map 1] failed as task task_1563258949712_0034_51_00_000004 failed after vertex succeeded.]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
INFO  : Completed executing command(queryId=hive_20190716163641_303daa3b-62b4-499a-9d3a-ab644e77c739); Time taken: 26.412 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00Vertex re-running, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00Vertex failed, vertexName=Map 1, vertexId=vertex_1563258949712_0034_51_00, diagnostics=[Vertex vertex_1563258949712_0034_51_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE, Vertex vertex_1563258949712_0034_51_00 [Map 1] failed as task task_1563258949712_0034_51_00_000004 failed after vertex succeeded.]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0 (state=08S01,code=2)

Total cluster memory is 384 GB with 56 cores and LLAP queue is allocated with 45% resources in yarn queue manager.

I have tweaked many setting after reading countless articles but none of them are helping me to resolve the issue.

A little help or hint towards right direction would be great.


1 REPLY 1

Re: Hive LLAP Vertex failure

Contributor

@ateeq ashraf,


Are you using a cache, can you check if the cache is full?