question Re: Is using data compression is better practice while storing data in hdfs? in Archives of Support Questions (Read Only)

Is using data compression is better practice while storing data in hdfs?

rushikeshdeshmu — Thu, 18 Feb 2016 17:02:18 GMT

Hi,

Is using data compression is better practice while storing data in hdfs?, I have found many different views regarding this. Can anyone explain whether it is good practice to use compression or it should be avoided?

Re: Is using data compression is better practice while storing data in hdfs?

kgopal — Thu, 18 Feb 2016 17:12:14 GMT

Hi Rushikesh,

Hadoop jobs are data intensive, compressing data can speed up the I/O operations

- MapReduce jobs are almost always I/O bound

Compressed data can save storage space and speed up data transfers across the network - Capital allocation for hardware can go further

Reduced I/O and network load can bring significant performance improvements

- MapReduce jobs can finish faster overall

On the other hand, CPU utilization and processing time increases during compression and decompression

- Understanding the tradeoff is important for MapReduce pipeline's overall performance

Re: Is using data compression is better practice while storing data in hdfs?

kgopal — Thu, 18 Feb 2016 17:12:34 GMT

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_hdfs_admin_tools/content/ch04.html

Re: Is using data compression is better practice while storing data in hdfs?

rushikeshdeshmu — Sat, 20 Feb 2016 21:39:37 GMT

@Karthik Gopal, thanks for sharing this link.