Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Parquet index page for Impala

Solved Go to solution

Parquet index page for Impala

Rising Star

Hi dear experts!

 

could anybody know does parquet index page available for Impala(https://github.com/Parquet/parquet-format)? if so, where i can find more information about that?

 

thank you in advance!

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Parquet index page for Impala

Master Collaborator

Hi fil,

 

I'm afraid Impala currently does not take advantage of index pages when reading/writing data.

 

Index pages are part of the Parquet spec, so in theory, you shoud be able to write Parquet files with index pages (via some other tool) and have them be readable by Impala - but Impala will ignore the index pages.

 

We do plan on taking advantage of index pages and min/max/etc values, but we do not have a concrete target date for that feature yet.

 

Alex

 

3 REPLIES 3

Re: Parquet index page for Impala

Master Collaborator

Hi fil,

 

I'm afraid Impala currently does not take advantage of index pages when reading/writing data.

 

Index pages are part of the Parquet spec, so in theory, you shoud be able to write Parquet files with index pages (via some other tool) and have them be readable by Impala - but Impala will ignore the index pages.

 

We do plan on taking advantage of index pages and min/max/etc values, but we do not have a concrete target date for that feature yet.

 

Alex

 

Re: Parquet index page for Impala

Rising Star
thanks for your comment!
Highlighted

Re: Parquet index page for Impala

Explorer

https://github.com/Parquet/parquet-format/blob/f7ab552f569df63bdb59f751d0dd36e826682739/src/thrift/p...

 

Index pages are declared in Parquet format, but not actually implemented.

See code above. 

 

Don't have an account?
Coming from Hortonworks? Activate your account here