question Re: Is there a way to change compression level in Gzip in GzipCodec ? in Archives of Support Questions (Read Only)

Is there a way to change compression level in Gzip in GzipCodec ?

ravi1 — Fri, 16 Sep 2022 09:46:34 GMT

Looking at the code, it looks like GZipCodec uses Deflater.DEFAULT_COMPRESSION. Is there a way to tweak compression levels of Gzip for mapreduce output?

Re: Is there a way to change compression level in Gzip in GzipCodec ?

deepesh1 — Thu, 29 Oct 2015 04:38:20 GMT

Looking at HADOOP-5879 it seems you can.

Re: Is there a way to change compression level in Gzip in GzipCodec ?

nsabharwal — Thu, 29 Oct 2015 05:36:16 GMT

@ravi@hortonworks.com

Ravi, This is interesting find

Re: Is there a way to change compression level in Gzip in GzipCodec ?

ravi1 — Thu, 29 Oct 2015 05:48:38 GMT

Interesting find. It is still not clear how we can send this configuration for mapreduce output. Right now, we just specify Codec.

Re: Is there a way to change compression level in Gzip in GzipCodec ?

nsabharwal — Thu, 29 Oct 2015 17:19:03 GMT

I agree. I am trying to reach out to the contributors on that thread. I found this article

Let me know if that helps to being with the customization on compression.

@ravi@hortonworks.com

Re: Is there a way to change compression level in Gzip in GzipCodec ?

ravi1 — Mon, 02 Nov 2015 12:24:32 GMT

Going through the code, I found the way to set GzipCodec compression level.

Right now, GzipCodec supports BEST_COMPRESSION, BEST_SPEED, NO_COMPRESSION and DEFAULT. Gzip itself supports 1-9 compression levels. But GzipCodec can use only BEST_COMPRESSION(9), BEST_SPEED(1) and DEFAULT (6). You can set them by passing zlib.compress.level = BEST_SPEED or BEST_COMRPESSION.

However, looking at numbers in our tests, a compression level of 4 seems to be best compression per CPU time. This is not possible right now to set level 4.

P.S. HDP 2.4 onwards, you can add other compression levels like 4. https://issues.apache.org/jira/browse/HADOOP-12794 has more details.

Re: Is there a way to change compression level in Gzip in GzipCodec ?

nsabharwal — Mon, 02 Nov 2015 19:58:26 GMT

Thanks @ravi@hortonworks.com