I am trying to run simple spark job using oozie workflow scheduler. And I am getting error as System memory 202375168 must be at least 4.718592E8. Please use a larger heap size. i have assigned 5 GB to my HDP sandbox on Virtual box. I have created a spark jar on my local machine and uploaded the jar to HDP sandbox. My workflow.xml looks like as below. <workflow-app name="samplespark-wf" xmlns="uri:oozie:workflow:0.4"> <start to="sparkjob"/>
<action name="sparkjob">
<spark xmlns="uri:oozie:spark-action:0.1"> <job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node> <master>local[1]</master>
<name>Spark Test</name>
<class>main.scala.RDDscala.RDD1</class> <jar>${nameNode}/spark_oozie_action/sparkrdd_2.11-0.0.1.jar</jar>
<spark-opts>--driver-memory 5g --num-executors 1</spark-opts> </spark>
<ok to="end"/>
<error to="fail"/> </action>
<kill name="fail">
<message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<kill name="fail-output">
<message>Incorrect output, expected [Hello Oozie] but was [${wf:actionData('shell-node')['my_output']}]</message>
</kill>
<end name="end"/>
</workflow-app> This program runs fine on local system command prompt with below mentioned command spark-submit --class main.scala.RDDscala.RDD2 --master local target\scala-2.11\sparkrdd_2.11-0.0.1.jar Any help would be appreciated
... View more
