To add to @emaxwell > If I am only concerned with performance
The bigger win is being able to skip decompressing blocks entirely - if you have hive.optimize.index.filter=true, that will kick in. > a few items in a where clause That's where the ORC indexes matter - if you have orc.create.index=true & orc.bloom.filter.columns contain those columns specifically (using "*" is easy, but slows down ETL when tables are wider and the measures are random) Clustering & Sorting on the most common column in the filter there can give you 2-3 magnitudes of performance (sorting specifically, because the min/max are stored at the footer of a file - this naturally happens for most ETL for date/timestamp columns, but for something randomly distributed like a location id, that is a big win). See for example, this ETL script
https://github.com/t3rmin4t0r/all-airlines-data/blob/master/ddl/orc.sql#L78
... View more
