Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark 1.6.1 with Orc Files

Highlighted

Spark 1.6.1 with Orc Files

New Contributor

Hi,

I am working with Spark 1.6.1 and hive 1.2.0.

I have Hive managed Table stored in Orc format.

I am trying to query this table and make sure predicates are pushed down,

Option#1) using Spark Sql, and I am using Spark Thrift server

when i look at the explain plan, I always see HiveTableScan. and I believe this is reading entire data. How can i make sure predicates are pushed down.

Option#2) using Hive, and I am using Hive Thrift server

and here all the time, I see a Map Reduce Job being triggered. how can i validate predicate pushdown is happening.

I want to make sure data is read in an optimized way.

thanks for your help

Regards

1 REPLY 1
Highlighted

Re: Spark 1.6.1 with Orc Files

Don't have an account?
Coming from Hortonworks? Activate your account here