Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to set min size avro files hive on spark

How to set min size avro files hive on spark

New Contributor

I tried to set the a minimun size in the avro output like in mr with hive on spark.

 

Which setters i need ?

 

I try with:

set hive.merge.size.per.task=1073741824;

set hive.merge.smallfiles.avgsize=536870912;

set hive.merge.mapfiles=true;

set hive.merge.mapredfiles=true;

set hive.merge.sparkfiles=true;

set hive.execution.engine=spark;

set spark.dynamicAllocation.enabled=true;

set spark.master=yarn-cluster;

 

I changed these 

set hive.merge.mapfiles=true;

set hive.merge.mapredfiles=true;

To false but didnt work 

 

Which is wrong?

 

Thanks