Small file in hadoop

[#1]

Hi Team ,

As we have more then 14 million small file as per Cloudera Navigator as below :

14.8M Small files created in 
the last 30 days 
 

14.8M / 21.4M
 69.3% small files

We are doing partition of data per day wise can we increase it or any other suggestion is there to overcome the small file problem.

Thank You in Advance

[#2]

@cdl-support Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our Atlas experts @BennyZ and @mayank_gupta  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3 · Accepted solution]

Hi @cdl-support . You can refer to the below article and check if those help.

https://my.cloudera.com/knowledge/Issue-with-Small-Files-in-HDFS?id=308948

Using Hive : https://docs.cloudera.com/best-practices/latest/impala-performance/topics/bp-impala-avoiding-small-f...

[#4]

@cdl-support Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.