question Re: best Compression technique? in Archives of Support Questions (Read Only)

best Compression technique?

pmj — Fri, 16 Sep 2022 12:25:44 GMT

Hi, I have tables in hive. some are text format, some are orc... what are the best compression methods to compress the text file so that i can still query on the table?

Re: best Compression technique?

kgautam — Sat, 21 Oct 2017 01:25:24 GMT

Text file : As text files are also included , please choose a compression algorithm which support splits. (gz is a big No)
Orc : Orc does a block level compression hence always splittable .

Overall : Please use ZLIB, SNAPPY, lzo splliatble for compression.

*orc.compress : high level compression (one of NONE, ZLIB, SNAPPY)

Re: best Compression technique?

divakarreddy_a — Sat, 21 Oct 2017 01:39:46 GMT

@PJ

We are using ORC Snappy on production cluster and Here are few useful links on compression:

https://community.hortonworks.com/questions/4067/snappy-vs-zlib-pros-and-cons-for-each-compression.html

https://community.hortonworks.com/articles/49252/performance-comparison-bw-orc-snappy-and-zlib-in-h.html

Re: best Compression technique?

pmj — Sat, 21 Oct 2017 01:53:16 GMT

@kgautam

how about parquet?

Re: best Compression technique?

kgautam — Sat, 21 Oct 2017 07:06:20 GMT

ORC and parquet both columnar format data storage and are competitors in terms of support an development.

Re: best Compression technique?

m_basavaraj — Sat, 21 Oct 2017 12:48:51 GMT

ORC is best option with in hive and Parquet is best option across Hadoop ecosystem.