0: jdbc:hive2://serv01:2181,ks-> select sum(destination),messagetype from t1.cdr where hday='2017-09-14' group by messagetype;
INFO : Session is already open
INFO : Dag name: select sum(destination),messag...messagetype(Stage-1)
ERROR : Status: Failed
ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1511771679762_0301_2_00, diagnostics=[Vertex vertex_1511771679762_0301_2_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: cdr initializer failed, vertex=vertex_1511771679762_0301_2_00 [Map 1], java.lang.RuntimeException: serious problem
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1277)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1304)
at org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat.getSplits(BucketizedHiveInputFormat.java:141)
at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:448)
at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:300)
at org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:122)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:273)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:266)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: delta_16881612_29766798 does not start with base_
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1254)
... 16 more
Caused by: java.lang.IllegalArgumentException: delta_16881612_29766798 does not start with base_
at org.apache.hadoop.hive.ql.io.AcidUtils.parseBase(AcidUtils.java:190)
at org.apache.hadoop.hive.ql.io.AcidUtils.parseBaseBucketFilename(AcidUtils.java:221)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.callInternal(OrcInputFormat.java:804)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.access$600(OrcInputFormat.java:747)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator$1.run(OrcInputFormat.java:772)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator$1.run(OrcInputFormat.java:769)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:769)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:747)
... 4 more
]
Error occur if delta_* present : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
[serv01]$ hdfs dfs -ls /warehouse/t1/cdr/hday=2017-09-14
Found 15 items
drwxrwxrwx - hive hdfs 0 2017-09-16 11:29 /warehouse/t1/cdr/hday=2017-09-14/base_16881497
drwxrwxrwx - hive hdfs 0 2017-10-21 18:42 /warehouse/t1/cdr/hday=2017-09-14/delta_16881612_29766798
drwxr-xr-x - hive hdfs 0 2017-10-22 17:48 /warehouse/t1/cdr/hday=2017-09-14/delta_30628231_30628231_0000
drwxr-xr-x - hive hdfs 0 2017-10-26 18:06 /warehouse/t1/cdr/hday=2017-09-14/delta_33418590_33418590_0000
drwxr-xr-x - hive hdfs 0 2017-10-27 16:23 /warehouse/t1/cdr/hday=2017-09-14/delta_33540229_33540229_0000
drwxr-xr-x - hive hdfs 0 2017-10-27 16:33 /warehouse/t1/cdr/hday=2017-09-14/delta_33541305_33541305_0000
drwxr-xr-x - hive hdfs 0 2017-10-31 12:40 /warehouse/t1/cdr/hday=2017-09-14/delta_34016509_34016509_0000
drwxr-xr-x - hive hdfs 0 2017-10-31 13:30 /warehouse/t1/cdr/hday=2017-09-14/delta_34025608_34025608_0000
drwxr-xr-x - hive hdfs 0 2017-10-31 14:19 /warehouse/t1/cdr/hday=2017-09-14/delta_34033668_34033668_0000
drwxr-xr-x - hive hdfs 0 2017-11-01 21:38 /warehouse/t1/cdr/hday=2017-09-14/delta_34219785_34219785_0000
drwxr-xr-x - hive hdfs 0 2017-11-02 11:20 /warehouse/t1/cdr/hday=2017-09-14/delta_34292833_34292833_0000
drwxr-xr-x - hive hdfs 0 2017-11-10 09:52 /warehouse/t1/cdr/hday=2017-09-14/delta_35449030_35449030_0000
drwxr-xr-x - hive hdfs 0 2017-11-10 13:07 /warehouse/t1/cdr/hday=2017-09-14/delta_35472185_35472185_0000
drwxr-xr-x - hive hdfs 0 2017-11-13 19:07 /warehouse/t1/cdr/hday=2017-09-14/delta_35944544_35944544_0000
drwxr-xr-x - hive hdfs 0 2017-11-21 12:37 /warehouse/t1/cdr/hday=2017-09-14/delta_36820930_36820930_0000
Workaround:
ALTER TABLE .. SET TBLPROPERTIES ('compactorthreshold.hive.compactor.delta.num.threshold'='1') - and wait done compactors working
set hive.execution.engine=mr;
Question :
exist any other workaround for run select with TEZ ?
