question Small file in hadoop in Support Questions

Small file in hadoop

cdl-support — Thu, 20 Jul 2023 10:16:47 GMT

Hi Team ,

As we have more then 14 million small file as per Cloudera Navigator as below :

14.8M Small files created in
the last 30 days

14.8M / 21.4M
69.3% small files

We are doing partition of data per day wise can we increase it or any other suggestion is there to overcome the small file problem.

Thank You in Advance

Re: Small file in hadoop

DianaTorres — Thu, 20 Jul 2023 20:24:01 GMT

@cdl-support Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our Atlas experts @BennyZ and @mayank_gupta who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Re: Small file in hadoop

rki_ — Fri, 21 Jul 2023 11:11:34 GMT

Hi @cdl-support . You can refer to the below article and check if those help.

https://my.cloudera.com/knowledge/Issue-with-Small-Files-in-HDFS?id=308948

Using Hive : https://docs.cloudera.com/best-practices/latest/impala-performance/topics/bp-impala-avoiding-small-files.html

Re: Small file in hadoop

DianaTorres — Tue, 25 Jul 2023 20:02:11 GMT

@cdl-support Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.