CONTROL SIZE OUTPUT FILE SIZE WITHOUT ADDING MERGE PROPERTY IN HIVE

[#1]

hI,

I am trying to reduce the small files but adding merge property affects the performance; since seperate job is triggered for this merge. Is there any way, to control the size of output file by mapper or reducer?

Thanks in advance!\

Mithuun

[#2 · Accepted solution]

Hello Mithun

Having a merge step is definitely more full proof approach. Otherwise you will to know more of your data and distribution and set yourself. A first step would be hive.merge.smallfiles.avgsize that would add the extra step only of the average is not respected.

You can also set the number of reducers yourself either statically or dynamically based on the volume of data coming in and if you know your workload this will allow you to calculate the file output size roughly.

Seems like a trade off between a more generic approach with a merge step and a more granular approach in which you know your workload.

hope this helps