I am trying to see the impact of predicate pushdown in ORC files. But till now for almost every query, data read from disk is same as size if table (hence no pruning). I think maybe I am missing some Hive setting or Table property. I run Hive 0.13 and Hadoop 2.7.0-mapr-1506. ... View more

I created a ORC table A: create table A like B; alter table A set fileformat orc; insert into table A select * from B sort by age; But, When I run query, select id from A where age=60; The data read from HDFS is almost equal to the size of A. I expected lesser data due to predicate pushdown. Also, in reducer logs there is no mention of ORC. ... View more

I have a table A with a column age (string) in it. (table size is 74GB, Hive 0.13) Now I created a table B with same data as of A, but in ORC file format and also included sort by age while creating B.   Now when I run the query: select count(id) from X where age=25;   The data read from B table is same as size of B (I expected predicate pushdown) and also the time taken in A is almost equal to B. Theoretically, because of predicate pushdown in ORC, a lot of data must have been skipped thus saving a lot of time. I suspect the indexes created in orc file are not getting read. I have tried almost everything. But nothing works. Please help with this. ... View more

I have a table A, with age a column in it. I created a table B in orc with age column sorted. Now, select * from A where age=60; select * from B where age=60; Both reads same amount of data and no difference in time was observed. Please help me with this. ... View more

when I run any query with explain on my table A (Hive 0.13). Below is a snapshot of result after a simple select query. STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 is a root stage STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree: TableScan alias: temp_segments Statistics: Num rows: ****** Data size: ****** Basic stats: COMPLETE Column stats: NONE Select Operator expressions: column1 (type: string) outputColumnNames: _col0 Statistics: Num rows: ****** Data size: ****** Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: true Statistics: Num rows: ****** Data size: ****** Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: -1 The table is in ORC for sure (verified). But here it is showing that table is text file. Any idea why is this there? ... View more

I have a partitioned table A in sequence file. Now to want to create a partitioned parquet table B with schema and data same as A. What I have tried: create table A like B; alter table A set fileformat parquet; insert into table A partition(part) select*from B; This gives error (surprisingly works for fileformat orc) SemanticException Line 1:18 Exception while processing 'A': must specify an InputFormat class Now something like this also does not work. create table A stored as parquet partitioned by (part) like B; All my three requirements (parquet, partition, like B) just does not come in same line according to hive. How to create such a table? ... View more

My hadoop jobs fail if I set org.apache.hadoop.io.compress.DefaultCodec as my mapred.output.compression.codec in Hive 0.13. I am not able to find reasons. ... View more

So I compressed my table in hive using snappy compression and it did get compress. The size was reduced. But when i run hadoop fs -lsr /hive/user.db/table_name, I see no file extensions with .snappy. I want to know if they really were snappy compressed or not? ... View more

After converting a 10TB table A to 6 TB table B using ORC compression, I expected to see faster reads too. But even after firing numerous JOIN and SELECT and COUNT queries, either the performance is same or worse in ORC. What can be the reason for it? ... View more