Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

schema incompatible issue in impala

avatar
New Contributor

We are facing incompatible schema issue when there is change in file schema (when new fields being added to file) while querying on Impala but it works fine on Beeline. Please help us in resolving this issue.

4 REPLIES 4

avatar
Super Guru
What's the issue in impala? Impala don't see the new columns? Did you update table to include the new columns through Beeline? Have you validated metadata in Impala?

avatar
New Contributor

we altered the table from beeline.we see the new column in the table structure but we arent able to query the hive table through impala but the same query works from beeline. yes. we have invalidated the metadata in impala .

avatar
Super Guru
Hi Ullas,

Can you please share the DDL of the table from both Impala and Hive and also the ALTER command you used to update schema?

Thanks

avatar
New Contributor

The solution to your issue likely depends on what type of files back your table, but if you are using parquet, this option is probably what you are looking for:

 

 

 

set PARQUET_FALLBACK_SCHEMA_RESOLUTION=name;

 

https://www.cloudera.com/documentation/enterprise/5-8-x/topics/impala_parquet.html#parquet_schema_ev...

 

 

The issue is that by default, impala expects every entry in the parquet schema to be at the same ordinal position. If you add to your schema anywhere but the end impala starts throwing errors. The option above makes impala flexible about the ordinal positions within the parquet files.