schema incompatible issue in impala

[#1]

We are facing incompatible schema issue when there is change in file schema (when new fields being added to file) while querying on Impala but it works fine on Beeline. Please help us in resolving this issue.

[#2]

What's the issue in impala? Impala don't see the new columns? Did you update table to include the new columns through Beeline? Have you validated metadata in Impala?

[#3]

we altered the table from beeline.we see the new column in the table structure but we arent able to query the hive table through impala but the same query works from beeline. yes. we have invalidated the metadata in impala .

[#4]

Hi Ullas,

Can you please share the DDL of the table from both Impala and Hive and also the ALTER command you used to update schema?

Thanks

[#5]

The solution to your issue likely depends on what type of files back your table, but if you are using parquet, this option is probably what you are looking for:

set PARQUET_FALLBACK_SCHEMA_RESOLUTION=name;

https://www.cloudera.com/documentation/enterprise/5-8-x/topics/impala_parquet.html#parquet_schema_ev...

The issue is that by default, impala expects every entry in the parquet schema to be at the same ordinal position. If you add to your schema anywhere but the end impala starts throwing errors. The option above makes impala flexible about the ordinal positions within the parquet files.