Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Impala does not not support reserved words in complex types

Solved Go to solution

Impala does not not support reserved words in complex types

Explorer

I have been testing CDH5.5.0 and have noted that Impala does not like reserved words as field names in complex types. This seems strange as reserved words can be used as column names for ordinary columns. Hive does not impose the same restriction. Reserved words can be back-ticked where needed.

Does anybody know if this is by design or if this is an issue in Impala 2.3.0?

We are using Hive to create Parquet files with complex types. Sample to reproduce issue and error message below. In the case below the word 'replace' is reserved.


In Hive:

CREATE EXTERNAL TABLE MyTable (
  device_id STRING,
  added struct<name:string,version_name:string,version_code:int,`replace`:boolean>
)
STORED AS PARQUET
LOCATION '/tmp/impala/mytable'; 

 

In Hive:

INSERT OVERWRITE TABLE MyTable
SELECT  
  device_id,
  payload AS added
FROM Added where import_id = 106000;

 
In Impala:

SELECT * FROM MyTable limit 10;


Output:

AnalysisException: Failed to load metadata for table: 'mytable' CAUSED BY: TableLoadingException: Unsupported type 'struct<name:string,version_name:string,version_code:int,replace:boolean>' in column 'added' of table 'mytable'  


I0107 15:56:01.251721 21006 Frontend.java:818] analyze query SELECT * FROM MyTable limit 10
E0107 15:56:01.252320 21006 Analyzer.java:2212] Failed to load metadata for table: mytable
Unsupported type 'struct<name:string,version_name:string,version_code:int,replace:boolean>' in column 'added' of table 'mytable'
I0107 15:56:01.252908 21006 jni-util.cc:177] com.cloudera.impala.common.AnalysisException: Failed to load metadata for table: 'MyTable'
        at com.cloudera.impala.analysis.TableRef.analyze(TableRef.java:180)
        at com.cloudera.impala.analysis.Analyzer.resolveTableRef(Analyzer.java:512)
        at com.cloudera.impala.analysis.SelectStmt.analyze(SelectStmt.java:155)
        at com.cloudera.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:342)
        at com.cloudera.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:317)
        at com.cloudera.impala.service.Frontend.analyzeStmt(Frontend.java:827)
        at com.cloudera.impala.service.Frontend.createExecRequest(Frontend.java:856)
        at com.cloudera.impala.service.JniFrontend.createExecRequest(JniFrontend.java:147)
Caused by: com.cloudera.impala.catalog.TableLoadingException: Unsupported type 'struct<name:string,version_name:string,version_code:int,replace:boolean>' in column 'added' of table 'mytable'
        at com.cloudera.impala.catalog.IncompleteTable.loadFromThrift(IncompleteTable.java:111)
        at com.cloudera.impala.catalog.Table.fromThrift(Table.java:240)
        at com.cloudera.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:357)
        at com.cloudera.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:246)
        at com.cloudera.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:132)
        at com.cloudera.impala.service.Frontend.updateCatalogCache(Frontend.java:223)
        at com.cloudera.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:164)
        at ========.<Remote stack trace on catalogd>: com.cloudera.impala.catalog.TableLoadingException: Unsupported type 'struct<name:string,version_name:string,version_code:int,replace:boolean>' in column 'added' of table 'mytable'
        at com.cloudera.impala.catalog.Table.parseColumnType(Table.java:331)
        at com.cloudera.impala.catalog.HdfsTable.addColumnsFromFieldSchemas(HdfsTable.java:571)
        at com.cloudera.impala.catalog.HdfsTable.load(HdfsTable.java:1073)
        at com.cloudera.impala.catalog.TableLoader.load(TableLoader.java:84)
        at com.cloudera.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:232)
        at com.cloudera.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:229)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Impala does not not support reserved words in complex types

Master Collaborator

Hi!

 

I confirm that this is a problem in Impala 2.3. It stems from the fact that the type is stored as a string inside the Hive Metastore, and upon table loading, Impala parses the string into an internal representation. However, the types stored inside the Metastore are not quoted, so Impala fails to parse the reserved words.

 

Would you mind filing a JIRA for this https://issues.cloudera.org with the info you provided below and my response above?

 

Thanks!

 

Alex

6 REPLIES 6

Re: Impala does not not support reserved words in complex types

Master Collaborator

Hi!

 

I confirm that this is a problem in Impala 2.3. It stems from the fact that the type is stored as a string inside the Hive Metastore, and upon table loading, Impala parses the string into an internal representation. However, the types stored inside the Metastore are not quoted, so Impala fails to parse the reserved words.

 

Would you mind filing a JIRA for this https://issues.cloudera.org with the info you provided below and my response above?

 

Thanks!

 

Alex

Re: Impala does not not support reserved words in complex types

Explorer

Thank you Alex for your quick reply and confirmation!

 

I've created IMPALA-2820 to track this issue.

 

 

Re: Impala does not not support reserved words in complex types

Master Collaborator

Thanks for reporting this issue and filing a JIRA! Much appreciated ;)

Re: Impala does not not support reserved words in complex types

Explorer

What a coinsidence! I've reported the same issue 1 hour after you :-)

https://issues.cloudera.org/browse/IMPALA-2821

Highlighted

Re: Impala does not not support reserved words in complex types

Explorer

Ah, more information for the team to work with then! Let's hope for a solution.

Re: Impala does not not support reserved words in complex types

Master Collaborator

Yea :). Thanks everyone for your reports and JIRA contricutions!

Don't have an account?
Coming from Hortonworks? Activate your account here