Skip to main content
cdl-support
cdl-support
Bronze III
July 20, 2023
Solved

Small file in hadoop

Hi Team ,

 

As we have more then 14 million small file as per Cloudera Navigator as below :

 

14.8M Small files created in 
the last 30 days 
 


14.8M / 21.4M
 69.3% small files

 

We are doing partition of data per day wise can we increase it or any other suggestion is there to overcome the small file problem.

 

Thank You in Advance

 
 
 
rki_
rki_Answer
Gold ⭐️
July 21, 2023

Hi @cdl-support . You can refer to the below article and check if those help.

 

https://my.cloudera.com/knowledge/Issue-with-Small-Files-in-HDFS?id=308948

 

Using Hive : https://docs.cloudera.com/best-practices/latest/impala-performance/topics/bp-impala-avoiding-small-files.html

View in thread

3 replies

DianaTorres
Community Manager
DianaTorres
Community Manager
July 20, 2023

@cdl-support Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our Atlas experts @BennyZ and @mayank_gupta  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Translate
Regards, Diana Torres, Senior Community Moderator
rki_
Gold ⭐️
rki_Answer
Gold ⭐️
July 21, 2023

Hi @cdl-support . You can refer to the below article and check if those help.

 

https://my.cloudera.com/knowledge/Issue-with-Small-Files-in-HDFS?id=308948

 

Using Hive : https://docs.cloudera.com/best-practices/latest/impala-performance/topics/bp-impala-avoiding-small-files.html

Translate
DianaTorres
Community Manager
DianaTorres
Community Manager
July 25, 2023

@cdl-support Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.

Translate
Regards, Diana Torres, Senior Community Moderator
Terms & ConditionsAccessibility statement