Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive Table formats

Solved Go to solution

Hive Table formats

Explorer

Hello,

I am Hive-testbench (http://blog.moserit.com/benchmarking-hive) to test some queries. By default, using the ./tpcds-setup.sh 10 what is the file format will my hive tables have (since in hdfs they are listed with a .deflate extesion)? I think the best file formats for performance are either ORC orc parquet, how can i generate in those formats?

Thanks

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Hive Table formats

7 REPLIES 7
Highlighted

Re: Hive Table formats

Highlighted

Re: Hive Table formats

Note: Parquet is supported for LLAP but will not by cached.

Highlighted

Re: Hive Table formats

@mÁRIO Rodrigues

Deflate is not a format. But if the file is in compressed state then the extension of your file in HDFS will be mentioned as .defalte. As you stated ORC performance better during loading the table. Parquet and Avro also serves its own purpose. When I have tested a table with 3 billion records the time taken for loading a hive table with specific format were

ORC

Avro

Parquet. In ascending order of time taken. ORC being the least amount of time taken during loading. But if your file format is dynamic then its better to go with parquet/Avro.

Highlighted

Re: Hive Table formats

Explorer

So although it presents itself as .deflate, basicly it's orc? Spark queries can query .parquet files, it will be able to query in these files with deflate format?

Highlighted

Re: Hive Table formats

@mÁRIO Rodrigues

Yes even though if its expressed as .deflate its in ORC with compressed state. I thik you will be able to read the files through hive tables in Spark SQL but you cant use the underneath files in it as it is compressed. If you want to read the files then load the hive tables without any compression and then Spark can make use of that file underneath.

Highlighted

Re: Hive Table formats

@mÁRIO Rodrigues

Refer to blog for details

Highlighted

Re: Hive Table formats

Explorer

Thank you all for the answers!

Don't have an account?
Coming from Hortonworks? Activate your account here