question Re: Hive Table formats in Archives of Support Questions (Read Only)

Hive Table formats

mmlr_90 — Mon, 08 May 2017 07:40:16 GMT

Hello,

I am Hive-testbench (http://blog.moserit.com/benchmarking-hive) to test some queries. By default, using the ./tpcds-setup.sh 10 what is the file format will my hive tables have (since in hdfs they are listed with a .deflate extesion)? I think the best file formats for performance are either ORC orc parquet, how can i generate in those formats?

Thanks

Re: Hive Table formats

SQLShaw — Mon, 08 May 2017 08:36:07 GMT

@mÁRIO Rodrigues use https://github.com/hortonworks/hive-testbench. Default format is ORC.

Re: Hive Table formats

SQLShaw — Mon, 08 May 2017 08:37:39 GMT

Note: Parquet is supported for LLAP but will not by cached.

Re: Hive Table formats

balavignesh_nag — Mon, 08 May 2017 13:28:01 GMT

@mÁRIO Rodrigues

Deflate is not a format. But if the file is in compressed state then the extension of your file in HDFS will be mentioned as .defalte. As you stated ORC performance better during loading the table. Parquet and Avro also serves its own purpose. When I have tested a table with 3 billion records the time taken for loading a hive table with specific format were

ORC

Avro

Parquet. In ascending order of time taken. ORC being the least amount of time taken during loading. But if your file format is dynamic then its better to go with parquet/Avro.

Re: Hive Table formats

ssubhas — Mon, 08 May 2017 16:30:31 GMT

@mÁRIO Rodrigues

Refer to blog for details

Re: Hive Table formats

mmlr_90 — Tue, 09 May 2017 07:34:25 GMT

Thank you all for the answers!

Re: Hive Table formats

mmlr_90 — Tue, 09 May 2017 07:37:17 GMT

So although it presents itself as .deflate, basicly it's orc? Spark queries can query .parquet files, it will be able to query in these files with deflate format?

Re: Hive Table formats

balavignesh_nag — Tue, 09 May 2017 22:38:45 GMT

@mÁRIO Rodrigues

Yes even though if its expressed as .deflate its in ORC with compressed state. I thik you will be able to read the files through hive tables in Spark SQL but you cant use the underneath files in it as it is compressed. If you want to read the files then load the hive tables without any compression and then Spark can make use of that file underneath.