Reply
New Contributor
Posts: 1
Registered: ‎05-09-2017

How to set min size avro files hive on spark

I tried to set the a minimun size in the avro output like in mr with hive on spark.

 

Which setters i need ?

 

I try with:

set hive.merge.size.per.task=1073741824;

set hive.merge.smallfiles.avgsize=536870912;

set hive.merge.mapfiles=true;

set hive.merge.mapredfiles=true;

set hive.merge.sparkfiles=true;

set hive.execution.engine=spark;

set spark.dynamicAllocation.enabled=true;

set spark.master=yarn-cluster;

 

I changed these 

set hive.merge.mapfiles=true;

set hive.merge.mapredfiles=true;

To false but didnt work 

 

Which is wrong?

 

Thanks