Thanks for suggestion, but currently I am not using enforce option. I will try by including with this option and run again.
I am using below options,
hive.exec.dynamic.partition --> true
hive.exec.dynamic.partition.mode --> nonstrict
hive.execution.engine --> tez
But in the given link, about enforce option, they specifically mentioned,
Not needed in Hive
and I am using 188.8.131.52
I am pretty sure HDP 184.108.40.206 does not have Hive2.x and Hive 2.x is GA in future releases of HDP, probably from HDP 2.6 or later.
If you are okay with the solution provided, can you please upvote and accept the answer ? Thanks
Thanks , yes, it worked by setting hive.enforce.bucketing = true.
but I didn't understand the background how it affects while querying thru tez and MR?
I have the same issue here. Will test with setting hive.enforce.bucketing=true while inserting data. But does anyone know why this setting will help here?
Bucketing is supported for hive 2.x and above.
set hive.enforce.bucketing =
The main reason is that it allows the correct number of reducers and the cluster by column to be automatically selected based on the table. Otherwise, you would need to set the number of reducers to be the same as the number of buckets as in
set mapred.reduce.tasks = 256; and have a
CLUSTER BY ... clause in the select.
my hdp is 2.3 hive 1.2 sql union all itself
use tez and orc is right
bug use mr is 0
this is my ddl
CREATE TABLE `test.web` ( `id` string , `uid` string , `user_id` int ) PARTITIONED BY (`p_date` string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' COLLECTION ITEMS TERMINATED BY ',' MAP KEYS TERMINATED BY ':' LINES TERMINATED BY '\n' NULL DEFINED AS '' STORED AS ORC TBLPROPERTIES('orc.compress'='SNAPPY')
SELECT count(*) FROM ( SELECT id, user_id FROM test.web WHERE p_date = 20171129 AND user_id > 0 UNION ALL SELECT id, user_id FROM test.web WHERE p_date = 20171129 AND stat_id = 'adm' AND user_id > 0 ) a
hive 1.2 hive.enforce.bucketing default is true
Do need other parameters?