question Re: I am trying to split file size to 64mb in Archives of Support Questions (Read Only)

I am trying to split file size to 64mb

aashanna — Fri, 07 Jul 2017 00:14:57 GMT

set mapred.max.split.size=67108864; set mapred.min.split.size=1024; set hive.execution.engine=tez; insert overwrite table bdd.signal_hte partition(cvdt36_year,cvdt36_mon,cvdt36_day) select * from cv.signal_hte where cvdt36_year= "2015" and cvdt36_mon =05;

Re: I am trying to split file size to 64mb

ssubhas — Fri, 07 Jul 2017 00:22:21 GMT

@Akhil Reddy

For tez, you need to use below parameter to set min and max splits of data:

set tez.grouping.min-size=16777216;--16 MB min split
set tez.grouping.max-size=64000000;--64 GB max split

Increase min and max split size to reduce the number of mappers.

Re: I am trying to split file size to 64mb

aashanna — Fri, 07 Jul 2017 00:56:29 GMT

1003608529

still the size remain same

Re: I am trying to split file size to 64mb

aashanna — Fri, 07 Jul 2017 00:58:15 GMT

if I do it for one day record it is working If I do for entire one year it still remain same.

Re: I am trying to split file size to 64mb

aashanna — Fri, 07 Jul 2017 01:02:37 GMT

insert overwrite table Mynewtable partition(cvdt36_year,cvdt36_mon,cvdt36_day) select * from MainTable where cvdt36_year= "2015" and cvdt36_mon =05 and cvdt36_day=16;

If I run this query it is working fine

insert overwrite table Mynewtable partition(cvdt36_year,cvdt36_mon,cvdt36_day) select * from MainTable where cvdt36_year= 2015; it is not working can you suggest me with correct query.

Re: I am trying to split file size to 64mb

ssubhas — Fri, 07 Jul 2017 01:10:30 GMT

@Akhil Reddy

There is no syntax issue with query, could you please share the issue you are facing?