Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Oozie Shell action with Spark-Submit : arguments to Spark-submit

Oozie Shell action with Spark-Submit : arguments to Spark-submit

New Contributor

Hi,

I am running oozie shell action with Spark-submit. It's working fine.

8934-wofkflowxml.jpg

but i want to send some additional input arguments to my main class as (ALDBROUGH,NATIONAL,normalized application,2016-05-31,2016-06-15 10:15:00,SECONDLY,AVG).

How to configure these additional input arguments in workflow.xml ?

Please find the command for run Spark job below.

spark-submit --master yarn-cluster --queue MCP--conf spark.yarn.scheduler.heartbeat.interval-ms=1000  --deploy-mode cluster --executor-memory 3G --driver-memory 3g  --num-executors 1 --files /etc/spark/2.3.4.0-3485/0/hive-site.xml --class com.socgen.ctt.mercure.dataseries.NormalizedDataSeries /tmp/rchamaku/DLLBigdata-0.0.1-SNAPSHOT.jar "[ALDBROUGH,NATIONAL,normalized application,2016-05-31,2016-06-15 10:15:00,SECONDLY,AVG]
3 REPLIES 3

Re: Oozie Shell action with Spark-Submit : arguments to Spark-submit

New Contributor
@Rambabu Chamakuri

did it work? adding "<java-opt>-Dparam1=val1</java-opt>

<java-opt>-Dparam2=val2</java-opt>

<java-opt>-Dparam3=val3</java-opt>" to the shell action?

Re: Oozie Shell action with Spark-Submit : arguments to Spark-submit

New Contributor

Hi, I am running spark submit command with oozie workflow, but getting error Mainclass[org.apache.oozie.action.hadoop],exit code [1]

I just wanted to confirm, if i need to give the HDFS path of jar and keytab in spark submit

Thanks in advance!!

Highlighted

Re: Oozie Shell action with Spark-Submit : arguments to Spark-submit

New Contributor

Only job.properties should be your local path.

HDFS Path of the Jar and Keytab is required