Hello Alex, Thanks for your time. Finally I got the point where I was doing the mistake, solved but still lot of questions in the mind. Objective: Convert the existing text table to parquet table for performance gains The actions performed are as below hive> create table t2 like t1; # t1 is text formatted table hive> insert into table t2 select * from t1; hive>alter table t2 set fileformat=parquet Impala> Invalidate metadata t2; Impala> desc formatted t2; Result: error, failed to load metadata, invalidate may help message. After posting this query and did some mor work, found that even Hive was not able to describe the table. NOte: Here I see a problem, we are in Hive version 12, the command returned successful, but it is not able to describe the table. I really see this as major issue, because when we allow the command and made successful changes to metadata , but unable to load metadata then we are lost, we will not be able to do any thing for this table, neither alter to original format nor drop the table. Don't you see this as issue? Now I found for Hive 12 version I need to follow the below alter table <tablename>set serde '<parquet serde>'; alter table <tablename> set fileformat inputformat "<parquet inputformat class>" alter table <tablename> set fileformat outputformat "<parquet outputformat class>" Now the first problem is solved, of course not on the same table as t2, i did exerice on another new tables, so now I could describe the table both on Impala and Hive. but not able to read the data because as you mentioned the underlying data is csv not parquet. Here I forgot to reload the data so that the data can be converted to parquet while copying it to new table with new altered format. finally now the issue is solved and more over it was not impala issue. but still I have doubt, may be fundamental: alter table set fileformat will just change table metadata information, so do we see really such cases where user will just need to alter table properties to parquet or ORC. Not able think of such scenario. when user has to take care of underlying data formats, then a warning mentioning that, user should alter data file to the respective <fileformat> to read the data would really help, I think
... View more
