question ORC vs Parquet - When to use one over the other in Archives of Support Questions (Read Only)

ORC vs Parquet - When to use one over the other

awatson — Sat, 24 Oct 2015 21:06:02 GMT

Hi All,

While ORC and Parquet are both columnar data stores that are supported in HDP, I was wondering if there was additional guidance on when to use one over the other? Or things to consider before choosing which format to use?

Thanks,

Andrew

Re: ORC vs Parquet - When to use one over the other

nsabharwal — Mon, 19 Aug 2019 12:56:09 GMT

@awatson@hortownorks.com

This blog is very useful. I share it with customers and prospects link

This focus on efficiency leads to some impressive compression ratios. This picture shows the sizes of the TPC-DS dataset at Scale 500 in various encodings. This dataset contains randomly generated data including strings, floating point and integer data.

Very well written - link

One thing to Note: Parquet default compression is SNAPPY.

This is not official statement. Based on aggressive testing in one of the environments

ORC+Zlib has better performance than Paqruet + Snappy

Re: ORC vs Parquet - When to use one over the other

rtempleton — Mon, 26 Oct 2015 23:42:54 GMT

In my mind the two biggest considerations for ORC over Parquet are:

1. Many of the performance improvements provided in the Stinger initiative are dependent on features of the ORC format including block level index for each column. This leads to potentially more efficient I/O allowing Hive to skip reading entire blocks of data if it determines predicate values are not present there. Also the Cost Based Optimizer has the ability to consider column level metadata present in ORC files in order to generate the most efficient graph.

2. ACID transactions are only possible when using ORC as the file format.

Re: ORC vs Parquet - When to use one over the other

aervits — Wed, 03 Feb 2016 00:35:03 GMT

@Andrew Watson has this been resolved? Can you accept best answer or provide your own solution?