Hi Royles, I am using hiveql for creating the table, altering the table for adding new columns. Doing all the operations like msck repair table,add partition to table everything I am doing from hiveql only.Only we are reading table from sparksql. After reading your reply,I tried to create external table,do msck repair,alter table to add new columns everything from sparksql. I got the below results 1.No results from spark when reading data from table 2.No results from hive shell when reading table 3.If I see the tblproperties,parquet schema is not matching .So there are no results from hiveql and from spark The only solution which I am following till now is(for adding new columns to external tbls) 1.Drop and create table using hiveql from hiveshell with all columns(old + new) 2.add latest partition manually which has data for all new columns added so far apart from beginning creation of table from hiveshell 3.query table from spark.Then check for tblproperties and parquet schema should be reflecting and mapped with hive columns 4.If the schema is not matching like testData in parquet is reflecting as testdata in hive tblproperties then we will get null values form spark 5.If both the schemas are matching,then we can see results from spark 4.then do msck repair which is giving me results in both spark 2.2 and 2.3 But I feel there must be some other way of adding new columns instead of dropping table and recreating it.
... View more
