Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive vertex issue

Hive vertex issue

Contributor

I keep getting this vertex issue when I run big queries... Can anyone point me to the right direction to resolve this issue? I did set the following settings before I run the query...

SET hive.tez.container.size=10240;

SET hive.tez.java.opts=-Xmx8192m;

set tez.runtime.io.sort.mb=4096;

set tez.runtime.unordered.output.buffer.size-mb=1024;

set hive.exec.dynamic.partition=true;

set hive.exec.dynamic.partition.mode=nonstrict;

set hive.tez.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;

set hive.vectorized.execution.reduce.enabled;

set hive.vectorized.execution.reduce.enabled=false;

Query ID = hilton_20160801135150_163b614d-fcad-4719-ba17-c62f3236e375 Total jobs = 1 Launching Job 1 out of 1 -------------------------------------------------------------------------------- VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED -------------------------------------------------------------------------------- Map 1 FAILED -1 0 0 -1 0 0 Map 2 ERROR 1 0 0 1 0 0 Map 3 ERROR 2 0 0 2 0 0 Map 4 ERROR 2 0 0 2 0 0 Map 5 ERROR 1 0 0 1 0 0 Map 6 KILLED 2 0 0 2 0 0 Map 7 ERROR 1 0 0 1 0 0 -------------------------------------------------------------------------------- VERTICES: 00/07 [>>--------------------------] 0% ELAPSED TIME: 1470059520.00 s -------------------------------------------------------------------------------- Status: Failed Vertex failed, vertexName=Map 1, vertexId=vertex_1469150491698_0298_3_01, diagnostics=[Vertex vertex_1469150491698_0298_3_01 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: f initializer failed, vertex=vertex_1469150491698_0298_3_01 [Map 1], java.lang.IllegalArgumentException at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1307) at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1195) at java.util.concurrent.Executors.newFixedThreadPool(Executors.java:89) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:519) at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:447) at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:299) at org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:121) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:255) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:248) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:248) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:235) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ] Vertex killed, vertexName=Map 7, vertexId=vertex_1469150491698_0298_3_06, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1469150491698_0298_3_06 [Map 7] killed/failed due to:OTHER_VERTEX_FAILURE] Vertex killed, vertexName=Map 5, vertexId=vertex_1469150491698_0298_3_05, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1469150491698_0298_3_05 [Map 5] killed/failed due to:OTHER_VERTEX_FAILURE, Invalid event V_INPUT_DATA_INFORMATION on vertex Map 5 with vertexId vertex_1469150491698_0298_3_05 at current state KILLED] Vertex killed, vertexName=Map 4, vertexId=vertex_1469150491698_0298_3_04, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1469150491698_0298_3_04 [Map 4] killed/failed due to:OTHER_VERTEX_FAILURE, Invalid event V_INPUT_DATA_INFORMATION on vertex Map 4 with vertexId vertex_1469150491698_0298_3_04 at current state KILLED] Vertex killed, vertexName=Map 3, vertexId=vertex_1469150491698_0298_3_03, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1469150491698_0298_3_03 [Map 3] killed/failed due to:OTHER_VERTEX_FAILURE, Invalid event V_INPUT_DATA_INFORMATION on vertex Map 3 with vertexId vertex_1469150491698_0298_3_03 at current state KILLED] Vertex killed, vertexName=Map 2, vertexId=vertex_1469150491698_0298_3_02, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1469150491698_0298_3_02 [Map 2] killed/failed due to:OTHER_VERTEX_FAILURE, Invalid event V_INPUT_DATA_INFORMATION on vertex Map 2 with vertexId vertex_1469150491698_0298_3_02 at current state KILLED] Invalid event V_INTERNAL_ERROR on Vertex vertex_1469150491698_0298_3_03 [Map 3] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1469150491698_0298_3_01, diagnostics=[Vertex vertex_1469150491698_0298_3_01 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: f initializer failed, vertex=vertex_1469150491698_0298_3_01 [Map 1], java.lang.IllegalArgumentException at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1307) at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1195) at java.util.concurrent.Executors.newFixedThreadPool(Executors.java:89) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:519) at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:447) at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:299) at org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:121) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:255) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:248) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:248) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:235) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ]Vertex killed, vertexName=Map 7, vertexId=vertex_1469150491698_0298_3_06, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1469150491698_0298_3_06 [Map 7] killed/failed due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Map 5, vertexId=vertex_1469150491698_0298_3_05, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1469150491698_0298_3_05 [Map 5] killed/failed due to:OTHER_VERTEX_FAILURE, Invalid event V_INPUT_DATA_INFORMATION on vertex Map 5 with vertexId vertex_1469150491698_0298_3_05 at current state KILLED]Vertex killed, vertexName=Map 4, vertexId=vertex_1469150491698_0298_3_04, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1469150491698_0298_3_04 [Map 4] killed/failed due to:OTHER_VERTEX_FAILURE, Invalid event V_INPUT_DATA_INFORMATION on vertex Map 4 with vertexId vertex_1469150491698_0298_3_04 at current state KILLED]Vertex killed, vertexName=Map 3, vertexId=vertex_1469150491698_0298_3_03, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1469150491698_0298_3_03 [Map 3] killed/failed due to:OTHER_VERTEX_FAILURE, Invalid event V_INPUT_DATA_INFORMATION on vertex Map 3 with vertexId vertex_1469150491698_0298_3_03 at current state KILLED]Vertex killed, vertexName=Map 2, vertexId=vertex_1469150491698_0298_3_02, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1469150491698_0298_3_02 [Map 2] killed/failed due to:OTHER_VERTEX_FAILURE, Invalid event V_INPUT_DATA_INFORMATION on vertex Map 2 with vertexId vertex_1469150491698_0298_3_02 at current state KILLED]Invalid event V_INTERNAL_ERROR on Vertex vertex_1469150491698_0298_3_03 [Map 3]

5 REPLIES 5
Highlighted

Re: Hive vertex issue

@Dagmawi Mengistu

Can you please try with below settings, in addition to yours:

set hive.execution.engine=tez;
set hive.auto.convert.join=true;
set hive.auto.convert.join.noconditionaltask=true;
set hive.auto.convert.join.noconditionaltask.size=405306368;
set hive.vectorized.execution.enabled=true;
set hive.vectorized.execution.reduce.enabled = true;
set hive.cbo.enable=true;
set hive.compute.query.using.stats=true;
set hive.stats.fetch.column.stats=true;
set hive.stats.fetch.partition.stats=true;
set hive.merge.mapfiles = true; 
set hive.merge.mapredfiles=true; 
set hive.merge.size.per.task=134217728; 
set hive.merge.smallfiles.avgsize= 44739242; 
set mapreduce.job.reduce.slowstart.completedmaps=0.7
Highlighted

Re: Hive vertex issue

Explorer

I also faced similar issue. Can you please help in that - http://stackoverflow.com/questions/39547001/why-hive-staging-file-is-missing-in-aws-emr?

Highlighted

Re: Hive vertex issue

New Contributor

Hi,

I am facing the below issue when i run "insert overwrite" query in hive.

Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1,

Highlighted

Re: Hive vertex issue

New Contributor

Hi @Sireesh Kumar ,

I am also facing the same issue while processing more than 10-12 partitions. Did anyone get the solution.

Highlighted

Re: Hive vertex issue

Did anything get the solution?

Don't have an account?
Coming from Hortonworks? Activate your account here