question Re: Data Compression Doesn't work in ORC with SNAPPY Compression in Archives of Support Questions (Read Only)

Data Compression Doesn't work in ORC with SNAPPY Compression

balavignesh_nag — Thu, 23 Mar 2017 19:58:56 GMT

I have a hive managed partition table (4 partitions) which has 2TB of data and it is stored as ORC tables with no compression. Now I have created a duplicate table with ORC -- SNAPPY compression and inserted the data from old table into the duplicate table. I noticed that it took more loading time than usual I believe that's because of enabling the compression. Then i have checked the file size in duplicate table with snappy compression and it shows somewhere around 2.6TB. Verified the count of both the tables and it remains the same. Any idea why the difference in size even after enabling the snappy compression in ORC?

Re: Data Compression Doesn't work in ORC with SNAPPY Compression

deepesh1 — Thu, 23 Mar 2017 23:06:15 GMT

Are you sure that the ORC tables you created were with no compression. By default hive.exec.orc.default.compress is set to ZLIB, perhaps your original table is with zlib compression.

There are some interesting threads to read:

https://community.hortonworks.com/questions/4067/snappy-vs-zlib-pros-and-cons-for-each-compression.html

https://community.hortonworks.com/articles/49252/performance-comparison-bw-orc-snappy-and-zlib-in-h.html

Re: Data Compression Doesn't work in ORC with SNAPPY Compression

balavignesh_nag — Fri, 24 Mar 2017 00:01:05 GMT

Thanks @Deepesh. You are right default compression is ZLIB and that causes the difference in compression.