Created on 03-15-2015 04:33 PM - edited 09-16-2022 02:24 AM
I can't get parameter substitution to work on my impala sql script even when I have it being passed through beeline (using it via hive2 action on oozie):
Script [customs_jp.sql] content: ------------------------ -- -- -- ... CREATE TABLE customers LIKE PARQUET '${INPUT}/_metadata' STORED AS PARQUET; ..
------------------------ Parameters: ------------------------ INPUT=/user/etl/transfer/customers/output1 ------------------------ Beeline command arguments : -u jdbc:impala://localhost:21050/;auth=noSasl -n etl -p DUMMY -d org.apache.hive.jdbc.HiveDriver --hivevar INPUT=/user/etl/transfer/customers/output1 -f customers.sql -a delegationToken =================================================================
on stderr:
Connected to: Impala (version 2.1.2-cdh5) Driver: ImpalaJDBC (version 02.05.05.1007) Error: [Simba][JDBC](11975) Unsupported transaction isolation level: 4. (state=HY000,code=11975) No rows affected (0.145 seconds) No rows affected (0.009 seconds) No rows affected (0.043 seconds) Error: [Simba][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: TStatus(statusCode:ERROR_STATUS, sqlState:HY000, errorMessage:AnalysisException: URI path must be absolute: ${INPUT}/_metadata), Query: CREATE TABLE customsers LIKE PARQUET '${INPUT}/_metadata' STORED AS PARQUET. (state=HY000,code=500051)
i.e. ${INPUT} is not being expanded.... (also tried ${hivevar:INPUT} without success...);
I see this bug here which seems related https://issues.cloudera.org/browse/IMPALA-1067 , but discusses impala-shell rather than beeline....
Any ideas ?
Created 03-20-2015 04:56 PM
I'm afraid Impala currently does not support variable substitution. One possible workaround is described here:
https://www.safaribooksonline.com/library/view/getting-started-with/9781491905760/ch04.html
Search for "Variable Substitution"
Created 03-20-2015 04:56 PM
I'm afraid Impala currently does not support variable substitution. One possible workaround is described here:
https://www.safaribooksonline.com/library/view/getting-started-with/9781491905760/ch04.html
Search for "Variable Substitution"
Created 03-20-2015 05:29 PM
Thanks. Actually based on that very same paragraph I ended up integrating jinja2 in my pipeline and have now a very nice framework for generating dynamic workflows and make parameters available anywhere 🙂
Thanks once again.