Archives of Support Questions (Read Only)

This is an archived board for historical reference. Information and links may no longer be available or relevant
Announcements
This board is archived and read-only for historical reference. To ask a new question, please post a new topic on the appropriate active board.

Can --hivevar be used in beeline with impala scripts ?

avatar
Contributor

I can't get parameter substitution to work on my impala sql script even when I have it being passed through beeline (using it via hive2 action on oozie):

 

Script [customs_jp.sql] content: 
------------------------
--
--
--
...
CREATE TABLE customers LIKE PARQUET '${INPUT}/_metadata' STORED AS PARQUET;
..
------------------------ Parameters: ------------------------ INPUT=/user/etl/transfer/customers/output1 ------------------------ Beeline command arguments : -u jdbc:impala://localhost:21050/;auth=noSasl -n etl -p DUMMY -d org.apache.hive.jdbc.HiveDriver --hivevar INPUT=/user/etl/transfer/customers/output1 -f customers.sql -a delegationToken =================================================================

on stderr:

Connected to: Impala (version 2.1.2-cdh5)
Driver: ImpalaJDBC (version 02.05.05.1007)
Error: [Simba][JDBC](11975) Unsupported transaction isolation level: 4. (state=HY000,code=11975)
No rows affected (0.145 seconds)
No rows affected (0.009 seconds)
No rows affected (0.043 seconds)
Error: [Simba][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: TStatus(statusCode:ERROR_STATUS, sqlState:HY000, errorMessage:AnalysisException: URI path must be absolute: ${INPUT}/_metadata), Query: CREATE TABLE customsers LIKE PARQUET '${INPUT}/_metadata' STORED AS PARQUET. (state=HY000,code=500051)

i.e. ${INPUT} is not being expanded.... (also tried ${hivevar:INPUT} without success...);

 

I see this bug here which seems related https://issues.cloudera.org/browse/IMPALA-1067  , but discusses impala-shell rather than beeline....

 

Any ideas ?

 

1 ACCEPTED SOLUTION

avatar

I'm afraid Impala currently does not support variable substitution. One possible workaround is described here:

https://www.safaribooksonline.com/library/view/getting-started-with/9781491905760/ch04.html

 

Search for "Variable Substitution"

 

View solution in original post

2 REPLIES 2

avatar

I'm afraid Impala currently does not support variable substitution. One possible workaround is described here:

https://www.safaribooksonline.com/library/view/getting-started-with/9781491905760/ch04.html

 

Search for "Variable Substitution"

 

avatar
Contributor

Thanks. Actually based on that very same paragraph I ended up integrating jinja2 in my pipeline and have now a very nice framework for generating dynamic workflows and make parameters available anywhere 🙂 

 

Thanks once again.