oozie launcher fails for spark-submit action, Caused by: java.io.IOException: error=2, No such file or directory

[#1]

spark-submit works completely fine and executes the query successfully, but if the same is called from oozie workflow as a action. we get the below error. "No such file or directory".

Caused by: java.io.IOException: error=2, No such file or directory

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], main() threw exception, java.io.IOException: Cannot run program "./spark-submit" (in directory "/usr/hdp/current/spark2-client/bin"): error=2, No such file or directory
org.apache.oozie.action.hadoop.JavaMainException: java.io.IOException: Cannot run program "./spark-submit" (in directory "/usr/hdp/current/spark2-client/bin"): error=2, No such file or directory

[#2]

@Fasil Ahamed

Does all the Oozie client nodes and NodeManagers have spark2-client installed?

Thanks

Venkat

[#3]

Yes , oozie client and spark2 client installed on all the node manager nodes.

[#4]

Yes , oozie client and spark2 client installed on all the node manager nodes.

any update here, as we see the issue is intermittent & sometimes the sqoop action throws the following error "No such file or directory"?

Quick help would be much appreciated.

[#5]

can you please provide job.properties and workflow.xml files and also how the spark submit is called i.e. are we using Oozie spark action or are we calling the spark-submit from a shell action?

[#6]

job.properties.

NameNode=hdfs://pap-hadoop-1:8020

jobTracker=pap-hadoop-2:8032

queueName=default examplesRoot=examplesoozie oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/core-services/workflow/Workflow.xml

Workflow.xml

<workflow-app xmlns="uri:oozie:workflow:0.1" name="pySparkTest">

<start to="spark_start"/> <action name="spark_start">

<java>

<job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node>

<configuration>

<property>

<name>mapred.queue.name</name>

<value>default</value>

</property>

<property>

<name>mapreduce.reduce.memory.mb</name>

<value>1024</value>

</property>

<property>

<name>mapreduce.map.memory.mb</name>

<value>1024</value>

</property>

<property>

<name>oozie.launcher.mapreduce.map.memory.mb</name>

<value>1024</value>

</property>

<property>

<name>oozie.launcher.yarn.app.mapreduce.am.resource.mb</name>

<value>1024</value>

</property>

</configuration>

<main-class>spark.SparkLauncher</main-class>

<arg>./spark-submit</arg>

<arg>--master</arg>

<arg>yarn</arg>

<arg>--deploy-mode</arg>

<arg>client</arg>

<arg>sparktest.py</arg>

<arg>scripts/sparktest.py</arg>

</java>

<ok to="end"/>

<error to="fail"/>

</action> <kill name="fail"> <message>Workflow Killed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill>

<end name="end"/>

</workflow-app>

[#7]

It is not right way to execute spark action. It is failing because it is not able to find spark-submit command in cache directory while launching it.

Instead use spark action:

Link - https://gist.github.com/rajkrrsingh/71f43afaac098428dc614d50ca0293ac

[#8]

Thanx @schhabra,

While i give a try with the process of shared link, i would like to highlight a note that the issue is intermittent & sometimes the same issue occurs for sqoop action "No Such file or directory" . is oozie missing on any jars or expecting any specific property?

[#9]

Hi All,

I figured out the fix , oozie is running the shell actions which is making YARN to call the necessary programs underneath. Since one of my Host did not have the spark-submit programme installed(whenever there was a call from YARN on that node, it used to fail.). I configured the programme on the missing host.

Now everything is running smooth. :)