Created 02-22-2016 01:58 PM
Thanks to all who participated in this thread. Check out the Community Knowledge article we created based on it. :)
Created on 09-01-2016 03:40 AM - edited 09-01-2016 05:25 AM
2016-09-01 12:32:17,252 WARN org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[******] USER[****] GROUP[-] TOKEN[] APP[PV3] JOB[0000050-160512133914543-oozie-oozi-W] ACTION[0000050-160512133914543-oozie-oozi-W@shell-7170] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], main() threw exception, Cannot run program "shell-impala-invalidate.sh" (in directory "/data/4/yarn/nm/usercache/****/appcache/application_1463053085953_30120/container_e49_1463053085953_30120_01_000002"): error=2, No such file or directory 2016-09-01 12:32:17,252 WARN org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[******] USER[****] GROUP[-] TOKEN[] APP[PV3] JOB[0000050-160512133914543-oozie-oozi-W] ACTION[0000050-160512133914543-oozie-oozi-W@shell-7170] Launcher exception: Cannot run program "shell-impala-invalidate.sh" (in directory "/data/4/yarn/nm/usercache/****/appcache/application_1463053085953_30120/container_e49_1463053085953_30120_01_000002"): error=2, No such file or directory java.io.IOException: Cannot run program "shell-impala-invalidate.sh" (in directory "/data/4/yarn/nm/usercache/*******/appcache/application_1463053085953_30120/container_e49_1463053085953_30120_01_000002"): error=2, No such file or directory
I tried to follow the tutorial, but for some reasons i get the following error.
My workflow.xml
<workflow-app name="shell-impala-invalidate-wf" xmlns="uri:oozie:workflow:0.4"> <start to="shell-impala-invalidate"/> <action name="shell-impala-invalidate"> <shell xmlns="uri:oozie:shell-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> </configuration> <exec>shell-impala-invalidate.sh</exec> <file>shell-impala-invalidate.sh#shell-impala-invalidate.sh</file> <file>shell-impala-invalidate.sql#shell-impala-invalidate.sql</file> <file>****.keytab#****.keytab</file> </shell> <ok to="end"/> <error to="kill"/> </action> <kill name="kill"> <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="end"/> </workflow-app>
Shell:
#!/bin/bash LOG=/tmp/shell-impala-invalidate-$USER.log ls -alrtR > $LOG #This will show you all the files in the directory and their relative paths export PYTHON_EGG_CACHE=./myeggs /usr/bin/kinit -kt ***.keytab -V *** /usr/bin/klist -e >> $LOG hadoop fs -put $LOG /tmp #put the log file in HDFS to find it easily impala-shell -f shell-impala-invalidate.sql
I feel a bit of lost now. Could any of you please give me a push, what is wrong?
Thanks
Created 09-02-2016 02:51 AM
Hello,
Your problem is not linked to the impala scheduler, but to the shell. In fact oozie cannot find your shell file.
1.What are the permission on the file? does oozie has acces?
shell-impala-invalidate.sh
2.What are the premission to the folder?
/data/4/yarn/nm/usercache/*******/appcache/application_1463053085953_30120/container_e49_1463053085953_30120_01_000002
(this folder is on one of your workers)
Alina
Created 09-02-2016 04:30 AM
Hi Alina,
First attempt:
I placed everything into my own library user / username, and gave the same permissions as in the attached picture.
Second attempt:
All the files were placed into the oozie workspace, (user/hue/oozie/workspaces/...), with the above permissions but the log message is still the same.
2.:
Can I check the permission to this folder through the webUI?
Created 09-15-2016 05:35 AM
Thyanks for the fruitful discussion , it saved my time. I have just made it working by making below changes
My workflow app xml :
<workflow-app name="shell-impala-invalidate-wf" xmlns="uri:oozie:workflow:0.4">
<start to="shell-impala-invalidate"/>
<action name="shell-impala-invalidate">
<shell xmlns="uri:oozie:shell-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>
<exec>${EXEC}</exec>
<file>${EXEC}#${EXEC}</file>
<file>${EXECQSCRIPT}#${EXECQSCRIPT}</file>
</shell>
<ok to="end"/>
<error to="kill"/>
</action>
<kill name="kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
where EXEC and EXECQSCRIPT are the path to the shell script and my sql file respectively.
In my shell script , I just need to add the following lines (I am not using secured environment here, so no Kerberos)
export PYTHON_EGG_CACHE= /home/python-egg-catch
impala-shell -f shellq.sql
where shellq.sql is my query file and python-egg-catch is the folder I have created with proper permission.
Thanks
Created on 11-03-2016 06:03 AM - edited 11-03-2016 06:06 AM
Hi - see this is quite an old thread, but was very useful to get me going. I've ended up writing a wrapper script to make the shell action behave like a HiveServer2 action from a parameter perspective. Here it is:
Usage Notes:
There are 4 mandatory parameters which must be in the correct ordinal position:
Parameter 5 onwards can be used for parameters in your SQL file that you want to be substituted in your reference SQL file. These take the form of <key>=<value> - and work in exactly the same way as HiveServer2 actions in that a parameter in the SQL file in the format "${<name>}" will be substituted with "<value>". In our example we have 2 additional parameters:
So the ${db_name} and ${table_name} tokens will be substituted with the supplied values:
Created 12-10-2018 02:07 AM