Member since
02-01-2018
2
Posts
0
Kudos Received
0
Solutions
02-27-2018
08:00 AM
Hello Mithun Having a merge step is definitely more full proof approach. Otherwise you will to know more of your data and distribution and set yourself. A first step would be hive.merge.smallfiles.avgsize that would add the extra step only of the average is not respected. You can also set the number of reducers yourself either statically or dynamically based on the volume of data coming in and if you know your workload this will allow you to calculate the file output size roughly. Seems like a trade off between a more generic approach with a merge step and a more granular approach in which you know your workload. hope this helps
... View more