https://community.cloudera.com/t5/Support-Questions/Compression-Zipping-of-old-unused-HDFS-files-in-Production/m-p/311268#M224563 <P>Hello there,<BR /><BR /></P><P>I understand your use-case to save up some HDFS space. Though I haven't tested zipping possibilities for hdfs level files[2].&nbsp;Alternately, you may consider reviewing HDFS Erasure Coding[1] if that suits your requirement:&nbsp;<BR />ErasureCoding in HDFS significantly reduces storage overhead while achieving similar or better fault tolerance through the use of parity cells (similar to RAID5). Prior to the introduction of EC, HDFS used 3x replication for fault tolerance exclusively, meaning that a 1GB file would use 3 GB of raw disk space. With EC, the same level of fault tolerance can be achieved using only 1.5 GB of raw disk space.<BR />Please refer the below article[1] for more insights on EC:</P><P>Ref[1]:&nbsp;<SPAN><A href="https://blog.cloudera.com/hdfs-erasure-coding-in-production/" target="_blank" rel="noopener">https://blog.cloudera.com/hdfs-erasure-coding-in-production/</A><BR /><BR /></SPAN></P><P><SPAN>[2] <A href="https://docs.cloudera.com/cloudera-manager/7.2.6/managing-clusters/topics/cm-choosing-configuring-data-compression.html" target="_blank" rel="noopener">https://docs.cloudera.com/cloudera-manager/7.2.6/managing-clusters/topics/cm-choosing-configuring-data-compression.html</A></SPAN></P> Wed, 10 Feb 2021 02:11:52 GMT shsingh 2021-02-10T02:11:52Z