Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

schema incompatible issue in impala

schema incompatible issue in impala

New Contributor

We are facing incompatible schema issue when there is change in file schema (when new fields being added to file) while querying on Impala but it works fine on Beeline. Please help us in resolving this issue.

4 REPLIES 4

Re: schema incompatible issue in impala

Guru
What's the issue in impala? Impala don't see the new columns? Did you update table to include the new columns through Beeline? Have you validated metadata in Impala?

Re: schema incompatible issue in impala

New Contributor

we altered the table from beeline.we see the new column in the table structure but we arent able to query the hive table through impala but the same query works from beeline. yes. we have invalidated the metadata in impala .

Re: schema incompatible issue in impala

Guru
Hi Ullas,

Can you please share the DDL of the table from both Impala and Hive and also the ALTER command you used to update schema?

Thanks

Re: schema incompatible issue in impala

New Contributor

The solution to your issue likely depends on what type of files back your table, but if you are using parquet, this option is probably what you are looking for:

 

 

 

set PARQUET_FALLBACK_SCHEMA_RESOLUTION=name;

 

https://www.cloudera.com/documentation/enterprise/5-8-x/topics/impala_parquet.html#parquet_schema_ev...

 

 

The issue is that by default, impala expects every entry in the parquet schema to be at the same ordinal position. If you add to your schema anywhere but the end impala starts throwing errors. The option above makes impala flexible about the ordinal positions within the parquet files.

 

Don't have an account?
Coming from Hortonworks? Activate your account here