Can --hivevar be used in beeline with impala scripts ?

balta — Fri, 16 Sep 2022 09:24:14 GMT

I can't get parameter substitution to work on my impala sql script even when I have it being passed through beeline (using it via hive2 action on oozie):

Script [customs_jp.sql] content: 
------------------------
--
--
--
...
CREATE TABLE customers LIKE PARQUET '${INPUT}/_metadata' STORED AS PARQUET;
..
------------------------

Parameters:
------------------------
  INPUT=/user/etl/transfer/customers/output1
------------------------

Beeline command arguments :
             -u
             jdbc:impala://localhost:21050/;auth=noSasl
             -n
             etl
             -p
             DUMMY
             -d
             org.apache.hive.jdbc.HiveDriver
             --hivevar
             INPUT=/user/etl/transfer/customers/output1
             -f
             customers.sql
             -a
             delegationToken

=================================================================

on stderr:

Connected to: Impala (version 2.1.2-cdh5)
Driver: ImpalaJDBC (version 02.05.05.1007)
Error: [Simba][JDBC](11975) Unsupported transaction isolation level: 4. (state=HY000,code=11975)
No rows affected (0.145 seconds)
No rows affected (0.009 seconds)
No rows affected (0.043 seconds)
Error: [Simba][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: TStatus(statusCode:ERROR_STATUS, sqlState:HY000, errorMessage:AnalysisException: URI path must be absolute: ${INPUT}/_metadata), Query: CREATE TABLE customsers LIKE PARQUET '${INPUT}/_metadata' STORED AS PARQUET. (state=HY000,code=500051)

i.e. ${INPUT} is not being expanded.... (also tried ${hivevar:INPUT} without success...);

I see this bug here which seems related https://issues.cloudera.org/browse/IMPALA-1067 , but discusses impala-shell rather than beeline....

Any ideas ?

Re: Can --hivevar be used in beeline with impala scripts ?

alex.behm — Fri, 20 Mar 2015 23:56:45 GMT

I'm afraid Impala currently does not support variable substitution. One possible workaround is described here:

https://www.safaribooksonline.com/library/view/getting-started-with/9781491905760/ch04.html

Search for "Variable Substitution"

Re: Can --hivevar be used in beeline with impala scripts ?

balta — Sat, 21 Mar 2015 00:29:50 GMT

Thanks. Actually based on that very same paragraph I ended up integrating jinja2 in my pipeline and have now a very nice framework for generating dynamic workflows and make parameters available anywhere 🙂

Thanks once again.

question Re: Can --hivevar be used in beeline with impala scripts ? in Archives of Support Questions (Read Only)

Can --hivevar be used in beeline with impala scripts ?

Re: Can --hivevar be used in beeline with impala scripts ?

Re: Can --hivevar be used in beeline with impala scripts ?