08-14-2016 03:00 AM - last edited on 08-15-2016 05:55 AM by cjervis
, in a similar way to Avro with avsc schema files which can be referenced in CREATE TABLE statements?
08-14-2016 03:24 AM
08-14-2016 06:07 AM
Thanks Harsh for confirming there is no external schema file concept in Parquet and for sharing the link for CREATE TABLE ... LIKE PARQUET ... syntax.
This seems to be specific to Impala however, is there a generic approach to use across a stack of tools including Spark, Pig, Hive as well as Impala (and with Spark and Pig not using HCatalog)?
08-14-2016 11:35 AM - edited 08-14-2016 11:41 AM
The whole support around Parquet is documented at http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_parquet.html
Impala's support for Parquet is ahead of Hive at this moment, while https://issues.apache.org/jira/browse/HIVE-8950 will help it catch up in future. In Hive you will still need to manually specify a column, but you may alternatively create the table in Impala and use it then in Hive.
Parquet's loader in Pig supports reading the schema off the file  , as does Spark's Parquet support . None of the eco system approaches use an external schema file as was the case with Avro storages.
 - https://github.com/Parquet/parquet-mr/blob/master/parquet-pig/src/main/java/parquet/pig/ParquetLoade...
 - https://github.com/Parquet/parquet-mr/blob/master/parquet-pig/src/test/java/parquet/pig/TestParquetL...
 - http://spark.apache.org/docs/latest/sql-programming-guide.html#parquet-files