Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

hive 2 select with group by error

Highlighted

hive 2 select with group by error

Contributor
0: jdbc:hive2://serv01:2181,ks-> select sum(destination),messagetype from   t1.cdr  where hday='2017-09-14' group by messagetype;
INFO  : Session is already open
INFO  : Dag name: select sum(destination),messag...messagetype(Stage-1)
ERROR : Status: Failed
ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1511771679762_0301_2_00, diagnostics=[Vertex vertex_1511771679762_0301_2_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: cdr initializer failed, vertex=vertex_1511771679762_0301_2_00 [Map 1], java.lang.RuntimeException: serious problem
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1277)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1304)
        at org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat.getSplits(BucketizedHiveInputFormat.java:141)
        at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:448)
        at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:300)
        at org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:122)
        at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:273)
        at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:266)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
        at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:266)
        at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: delta_16881612_29766798 does not start with base_
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1254)
        ... 16 more
Caused by: java.lang.IllegalArgumentException: delta_16881612_29766798 does not start with base_
        at org.apache.hadoop.hive.ql.io.AcidUtils.parseBase(AcidUtils.java:190)
        at org.apache.hadoop.hive.ql.io.AcidUtils.parseBaseBucketFilename(AcidUtils.java:221)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.callInternal(OrcInputFormat.java:804)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.access$600(OrcInputFormat.java:747)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator$1.run(OrcInputFormat.java:772)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator$1.run(OrcInputFormat.java:769)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:769)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:747)
        ... 4 more
]

Error occur if delta_* present :     <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

[serv01]$ hdfs dfs -ls  /warehouse/t1/cdr/hday=2017-09-14
Found 15 items
drwxrwxrwx   - hive hdfs          0 2017-09-16 11:29 /warehouse/t1/cdr/hday=2017-09-14/base_16881497
drwxrwxrwx   - hive hdfs          0 2017-10-21 18:42 /warehouse/t1/cdr/hday=2017-09-14/delta_16881612_29766798
drwxr-xr-x   - hive hdfs          0 2017-10-22 17:48 /warehouse/t1/cdr/hday=2017-09-14/delta_30628231_30628231_0000
drwxr-xr-x   - hive hdfs          0 2017-10-26 18:06 /warehouse/t1/cdr/hday=2017-09-14/delta_33418590_33418590_0000
drwxr-xr-x   - hive hdfs          0 2017-10-27 16:23 /warehouse/t1/cdr/hday=2017-09-14/delta_33540229_33540229_0000
drwxr-xr-x   - hive hdfs          0 2017-10-27 16:33 /warehouse/t1/cdr/hday=2017-09-14/delta_33541305_33541305_0000
drwxr-xr-x   - hive hdfs          0 2017-10-31 12:40 /warehouse/t1/cdr/hday=2017-09-14/delta_34016509_34016509_0000
drwxr-xr-x   - hive hdfs          0 2017-10-31 13:30 /warehouse/t1/cdr/hday=2017-09-14/delta_34025608_34025608_0000
drwxr-xr-x   - hive hdfs          0 2017-10-31 14:19 /warehouse/t1/cdr/hday=2017-09-14/delta_34033668_34033668_0000
drwxr-xr-x   - hive hdfs          0 2017-11-01 21:38 /warehouse/t1/cdr/hday=2017-09-14/delta_34219785_34219785_0000
drwxr-xr-x   - hive hdfs          0 2017-11-02 11:20 /warehouse/t1/cdr/hday=2017-09-14/delta_34292833_34292833_0000
drwxr-xr-x   - hive hdfs          0 2017-11-10 09:52 /warehouse/t1/cdr/hday=2017-09-14/delta_35449030_35449030_0000
drwxr-xr-x   - hive hdfs          0 2017-11-10 13:07 /warehouse/t1/cdr/hday=2017-09-14/delta_35472185_35472185_0000
drwxr-xr-x   - hive hdfs          0 2017-11-13 19:07 /warehouse/t1/cdr/hday=2017-09-14/delta_35944544_35944544_0000
drwxr-xr-x   - hive hdfs          0 2017-11-21 12:37 /warehouse/t1/cdr/hday=2017-09-14/delta_36820930_36820930_0000

Workaround:

ALTER TABLE  .. SET  TBLPROPERTIES ('compactorthreshold.hive.compactor.delta.num.threshold'='1')   - and wait done compactors working


set hive.execution.engine=mr;

Question :

exist any other workaround for run select with TEZ  ?








 


5 REPLIES 5
Highlighted

Re: hive 2 select with group by error

Contributor

HDP-2.6.3.0

Highlighted

Re: hive 2 select with group by error

Contributor

HDP-2.6.3.0

Highlighted

Re: hive 2 select with group by error

Contributor

Hi! all

any comment for me ?

Highlighted

Re: hive 2 select with group by error

Contributor

HDP-2.6.3.0

Highlighted

Re: hive 2 select with group by error

Contributor

HDP-2.6.3.0

Don't have an account?
Coming from Hortonworks? Activate your account here