Member since
03-14-2016
1
Post
3
Kudos Received
0
Solutions
03-14-2016
06:09 AM
3 Kudos
I recently tried spark action with Oozie and end up getting the following error: My job.properties file: ============================================================== jobTracker=qxnn:8032 master=local[2] queueName=default examplesRoot=spark-oozie oozie.use.system.libpath=true oozie.libpath=/user/oozie/share/lib oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}
workflow.xml: ================================================================ <workflow-app name="sparkJob" xmlns="uri:oozie:workflow:0.1"> <start to="spark-node"/> <action name="spark-node"> <spark xmlns="uri:oozie:spark-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.compress.map.output</name> <value>true</value> </property> </configuration> <master>yarn</master> <mode>cluster</mode> <name>Spark Example</name> <jar>/user/yesuser3/spark-oozie/hive2.py</jar> <spark-opts>--executor-memory 200M --num-executors 5</spark-opts> </spark> <ok to="end"/> <error to="spark-fail"/> </action> <kill name="spark-fail"> <message>Spark action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="end"/> </workflow-app>
Any help would be greatly appreciated. Thanks, SureshKumar
... View more
Labels:
- Labels:
-
Apache Oozie
-
Apache Spark