Created 05-23-2018 05:20 AM
Created 05-23-2018 05:24 AM
spark-submit works completely fine and executes the query successfully, but if the same is called from oozie workflow as a action. we get the below error. "No such file or directory".
Caused by: java.io.IOException: error=2, No such file or directory
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], main() threw exception, java.io.IOException: Cannot run program "./spark-submit" (in directory "/usr/hdp/current/spark2-client/bin"): error=2, No such file or directory org.apache.oozie.action.hadoop.JavaMainException: java.io.IOException: Cannot run program "./spark-submit" (in directory "/usr/hdp/current/spark2-client/bin"): error=2, No such file or directory
Created 05-23-2018 05:38 AM
Does all the Oozie client nodes and NodeManagers have spark2-client installed?
Thanks
Venkat
Created 05-23-2018 05:46 AM
Yes , oozie client and spark2 client installed on all the node manager nodes.
Created 06-11-2018 09:16 AM
Yes , oozie client and spark2 client installed on all the node manager nodes.
any update here, as we see the issue is intermittent & sometimes the sqoop action throws the following error "No such file or directory"?
Quick help would be much appreciated.
Created 05-23-2018 05:54 AM
can you please provide job.properties and workflow.xml files and also how the spark submit is called i.e. are we using Oozie spark action or are we calling the spark-submit from a shell action?
Created 05-23-2018 06:56 AM
job.properties.
NameNode=hdfs://pap-hadoop-1:8020
jobTracker=pap-hadoop-2:8032
queueName=default examplesRoot=examplesoozie oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/core-services/workflow/Workflow.xml
Workflow.xml
<workflow-app xmlns="uri:oozie:workflow:0.1" name="pySparkTest">
<start to="spark_start"/> <action name="spark_start">
<java>
<job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.queue.name</name>
<value>default</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>1024</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>1024</value>
</property>
<property>
<name>oozie.launcher.mapreduce.map.memory.mb</name>
<value>1024</value>
</property>
<property>
<name>oozie.launcher.yarn.app.mapreduce.am.resource.mb</name>
<value>1024</value>
</property>
</configuration>
<main-class>spark.SparkLauncher</main-class>
<arg>./spark-submit</arg>
<arg>--master</arg>
<arg>yarn</arg>
<arg>--deploy-mode</arg>
<arg>client</arg>
<arg>sparktest.py</arg>
<arg>scripts/sparktest.py</arg>
</java>
<ok to="end"/>
<error to="fail"/>
</action> <kill name="fail"> <message>Workflow Killed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill>
<end name="end"/>
</workflow-app>
Created 06-12-2018 07:43 PM
It is not right way to execute spark action. It is failing because it is not able to find spark-submit command in cache directory while launching it.
Instead use spark action:
Link - https://gist.github.com/rajkrrsingh/71f43afaac098428dc614d50ca0293ac
Created 06-13-2018 04:33 AM
Thanx @schhabra,
While i give a try with the process of shared link, i would like to highlight a note that the issue is intermittent & sometimes the same issue occurs for sqoop action "No Such file or directory" . is oozie missing on any jars or expecting any specific property?
Created 06-27-2018 11:35 AM
Hi All,
I figured out the fix , oozie is running the shell actions which is making YARN to call the necessary programs underneath. Since one of my Host did not have the spark-submit programme installed(whenever there was a call from YARN on that node, it used to fail.). I configured the programme on the missing host.
Now everything is running smooth. 🙂