Member since
11-03-2016
1
Post
0
Kudos Received
0
Solutions
11-03-2016
06:03 AM
Hi - see this is quite an old thread, but was very useful to get me going. I've ended up writing a wrapper script to make the shell action behave like a HiveServer2 action from a parameter perspective. Here it is: Usage Notes: There are 4 mandatory parameters which must be in the correct ordinal position: Script file name: e.g. test.sql **MUST BE REFERENCED AS A FILE IN THE ACTION The name of the keytab e.g. smithjon.keytab **MUST BE REFERENCED AS A FILE IN THE ACTION OR "NO_KEYTAB" IF NOT RUNNING IN A KERBERISED ENVIRONMENT The name of the principle in the keytab (who is requesting the query) e.g. smithjon@FOO.BAR.NET **USE NO_PRINCIPAL OR LEAVE BLANK IF NOT RUNNING IN A KERBERISED ENVIRONMENT The name of the impala service principle that will be executing the query e.g. fp-service-impala-p Parameter 5 onwards can be used for parameters in your SQL file that you want to be substituted in your reference SQL file. These take the form of <key>=<value> - and work in exactly the same way as HiveServer2 actions in that a parameter in the SQL file in the format "${<name>}" will be substituted with "<value>". In our example we have 2 additional parameters: db_name=bddqsit01p table_name=HELLO_IMPALA So the ${db_name} and ${table_name} tokens will be substituted with the supplied values: #!/bin/bash export PYTHON_EGG_CACHE=./myeggs #log the variables passed echo "script file: $1" echo "keytab file: $2" echo "local user: $3" echo "impala user: $4" # grab sql file into variable sql=`cat $1` echo "Redirecting raw sql to stderr to avoid stdout buffer issues" (>&2 echo "Received the following SQL") (>&2 echo $sql) #loop through all parameters and sed into the sql variable COUNTER=1 for TOKEN in $* do if [ $COUNTER -gt 4 ] ; then IFS='='; arrTOKEN=($TOKEN); unset IFS currKey=${arrTOKEN[0]} currValue=${arrTOKEN[1]} echo "$currKey=$currValue" sql=$(echo $sql| sed -e "s/\${$currKey}/$currValue/g") fi let COUNTER=COUNTER+1 done #kinit if required. Keytab name is the second parameter, user id the third if [ $2 != "NO_KEYTAB" ] ; then echo "Invoking with keytab" /usr/bin/kinit -kt $2 -V $3 echo "Redirecting parsed sql to stderr to avoid stdout buffer issues" (>&2 echo "Executing the following SQL") (>&2 echo $sql) impala-shell -q "$sql" -k -s $4 else echo "Invoking with no keytab" echo "Redirecting parsed sql to stderr to avoid stdout buffer issues" (>&2 echo "Executing the following SQL") (>&2 echo $sql) impala-shell -q "$sql" fi
... View more