Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]
Created on 03-15-2015 04:33 PM - edited 09-16-2022 02:24 AM
I can't get parameter substitution to work on my impala sql script even when I have it being passed through beeline (using it via hive2 action on oozie):
Script [customs_jp.sql] content:
------------------------
--
--
--
...
CREATE TABLE customers LIKE PARQUET '${INPUT}/_metadata' STORED AS PARQUET;
..
------------------------
Parameters:
------------------------
INPUT=/user/etl/transfer/customers/output1
------------------------
Beeline command arguments :
-u
jdbc:impala://localhost:21050/;auth=noSasl
-n
etl
-p
DUMMY
-d
org.apache.hive.jdbc.HiveDriver
--hivevar
INPUT=/user/etl/transfer/customers/output1
-f
customers.sql
-a
delegationToken
=================================================================
on stderr:
Connected to: Impala (version 2.1.2-cdh5)
Driver: ImpalaJDBC (version 02.05.05.1007)
Error: [Simba][JDBC](11975) Unsupported transaction isolation level: 4. (state=HY000,code=11975)
No rows affected (0.145 seconds)
No rows affected (0.009 seconds)
No rows affected (0.043 seconds)
Error: [Simba][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: TStatus(statusCode:ERROR_STATUS, sqlState:HY000, errorMessage:AnalysisException: URI path must be absolute: ${INPUT}/_metadata), Query: CREATE TABLE customsers LIKE PARQUET '${INPUT}/_metadata' STORED AS PARQUET. (state=HY000,code=500051)
i.e. ${INPUT} is not being expanded.... (also tried ${hivevar:INPUT} without success...);
I see this bug here which seems related https://issues.cloudera.org/browse/IMPALA-1067 , but discusses impala-shell rather than beeline....
Any ideas ?
Created 03-20-2015 04:56 PM
I'm afraid Impala currently does not support variable substitution. One possible workaround is described here:
https://www.safaribooksonline.com/library/view/getting-started-with/9781491905760/ch04.html
Search for "Variable Substitution"
Created 03-20-2015 04:56 PM
I'm afraid Impala currently does not support variable substitution. One possible workaround is described here:
https://www.safaribooksonline.com/library/view/getting-started-with/9781491905760/ch04.html
Search for "Variable Substitution"
Created 03-20-2015 05:29 PM
Thanks. Actually based on that very same paragraph I ended up integrating jinja2 in my pipeline and have now a very nice framework for generating dynamic workflows and make parameters available anywhere 🙂
Thanks once again.