Skip to main content
Mondi
Silver ⭐️⭐️⭐️
March 13, 2020
Solved

Does the 6.3.3 version Cloudera still experiencing small files issue?

Small and empty files are recurring on our current version of CDH Cluster. Does is still exist on 6.3.3 version?

StevenOD
StevenODAnswer
Platinum ⭐️
March 13, 2020

Hi @Mondi ,

 

Yes, small files can still cause an impact in CDH 6.3.3. This has nothing to do with the version of Cloudera but the way that the Namenode and HDFS interact when a lot of small files are stored in HDFS. Lots of small files create a lot of metadata that the Namenode must store and manage in memory. 

 

To understand more about the impact of small files in HDFS and how to manage this, please refer to this article:

 

https://blog.cloudera.com/small-files-big-foils-addressing-the-associated-metadata-and-application-challenges/

 

Regards,

Steve

2 replies

StevenOD
Platinum ⭐️
StevenODAnswer
Platinum ⭐️
March 13, 2020

Hi @Mondi ,

 

Yes, small files can still cause an impact in CDH 6.3.3. This has nothing to do with the version of Cloudera but the way that the Namenode and HDFS interact when a lot of small files are stored in HDFS. Lots of small files create a lot of metadata that the Namenode must store and manage in memory. 

 

To understand more about the impact of small files in HDFS and how to manage this, please refer to this article:

 

https://blog.cloudera.com/small-files-big-foils-addressing-the-associated-metadata-and-application-challenges/

 

Regards,

Steve

Translate
Mondi
MondiAuthor
Silver ⭐️⭐️⭐️
March 16, 2020

Thanks @StevenOD i'll check on this

Translate