We are using ParquetFileWriter to generate Parquet files and want to be able to query this in hive. So in hive we have it setup as an external table that is pointing to HDFS folder where parquet files are located. This all works great.
Next we tried to setup a partitioned tables so we changed writer to generate multiple folders based on the partition key Year. The external table in hive has been updated to have PartitionBy clause. We've also maually added partitions hive using alter table statement.
Now on querying the table we are getting incorrect results and it appers to be that the first parquet file loaded contents are returned for all the partitions. So if we query for year 2012 we get 5 records and we will get the same for 2013 and 2014. If we re-start hive shell and query directly for 2014 we correct result only for that partition and now subsequent queries all return the same data from 2014 partition.
We are using CDH 4.6 with Hive 0.10
Files weith created with following
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>0.10.0</version>
</dependency>
<dependency>
<groupId>com.twitter</groupId>
<artifactId>parquet-hive-bundle</artifactId>
<version>1.4.0</version>
</dependency>
create external table sinet_test
(
datatimestamp INT,
serverid INT
)
PARTITIONED BY (year String)
ROW FORMAT SERDE 'parquet.hive.serde.ParquetHiveSerDe'
STORED AS
INPUTFORMAT 'parquet.hive.DeprecatedParquetInputFormat'
OUTPUTFORMAT 'parquet.hive.DeprecatedParquetOutputFormat'
LOCATION '/tmp/sinet/writer';
Can someone please help look into this?
Thanks
Kanwal
