While executing a select sql on Hive we are seeing "Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1" error .

[#1]

While executing a select query when other queries are running in Hive we are facing below error . Can you some one suggest me what could be the reason for below error.

Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1470078643944_0469_5_00, diagnostics=[Vertex vertex_1470078643944_0469_5_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: se initializer failed, vertex=vertex_1470078643944_0469_5_00 [Map 1], org.apache.tez.dag.api.TezUncheckedException: Invalid max/min group lengths. Required min>0, max>=min. max: 4194304 min: 16777216

at org.apache.hadoop.mapred.split.TezMapredSplitsGrouper.getGroupedSplits(TezMapredSplitsGrouper.java:147)

at org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.group(SplitGrouper.java:89)

at org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:168)

at org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:138)

at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:159)

at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:273)

at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:266)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)

at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:266)

at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

]Vertex killed, vertexName=Map 2, vertexId=vertex_1470078643944_0469_5_01, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1470078643944_0469_5_01 [Map 2] killed/failed due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 3, vertexId=vertex_1470078643944_0469_5_02, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1470078643944_0469_5_02 [Reducer 3] killed/failed due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 4, vertexId=vertex_1470078643944_0469_5_03, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1470078643944_0469_5_03 [Reducer 4] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:3

Appreciated for your help in advance.

[#2]

Issue :-

Vertex failed, vertexName=Map 1, vertexId=vertex_1470078643944_0307_5_00, diagnostics=[Vertex vertex_1470078643944_0307_5_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: se initializer failed, vertex=vertex_1470078643944_0307_5_00 [Map 1], org.apache.tez.dag.api.TezUncheckedException: Invalid max/min group lengths. Required min>0, max>=min. max: 4194304 min: 16777216

Cause:-

In the client app ( Jmeter ) we are setting 3 parameters as below:-

set mapreduce.job.maps=64;

set tez.grouping.max-size=4194304;

set tez.grouping.min-size=2097152;

In Tez configuration , the max size is set as 1GB and min size as 16777216.

Now if you see the error , max shown as 4194304 which is that we exactly set through Jmeter and min is 16777216 which is the tez default parameter. So that means hive is taking the max from jmeter and min from tez default configuration and this is the cause of the issue.

[#3]

@sivakumar sudhakarannair girijakumari

This is a great finding. I did not realize that setting the ez.grouping.min-size at the session level will not override the global value. It should override. Maybe this is a bug which is a rare condition because nobody was thinking to override at the session level the tez.grouping.max-size so low that it would be lower than the tez.grouping.min-size set at global level. It is a small issue and it could be workarounded as I specified in my response below.

[#4 · Accepted solution]

@Mahipal Ramidi!

Ideally, keep Tez global settings as they are and set tez.grouping.max-size to a value that makes sense for the query you execute, always higher than the tez.grouping.min-size which you set globally. If your global tez.grouping_min-size is not low enough to allow you to set your session tez.grouping.max-size to a value higher than the global tez.grouping.min-size, you may want to change the global tez.grouping.min-size to a lower value to satisfy the condition. Low values of min and max create a lot of small tasks. Each task has allocated a container. Has such a lot of parallel tasks will do the work but it could also consume all resources of the cluster. This approach needs always a careful analysis of how many tasks created and resources used. Anyhow, mappers will chunk the input data to sizes between min and max and most likely there will be no impact on other jobs requiring a larger chunking. Your query seems to have a not so large data volume, but it requires a lot of parallelism to complete faster.

[#5]

@All I also similar issue. Can you please help me in that - http://stackoverflow.com/questions/39547001/why-hive-staging-file-is-missing-in-aws-emr?