Support Questions
Find answers, ask questions, and share your expertise

Add multiple options to spark.exeuctor.extraJavaOptions

New Contributor

In HUE UI, using Spark action widget, how to set up multiple options?



-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps


This does not work in "Options List" box

--conf spark.exeuctor.extraJavaOptions="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"



CDH 5.7.1 - Same issue, but for configuring app-specific log4j:


Working spark-submit command line: 

[--master yarn-cluster --files hdfs:/user/myao/config/ --conf "" --conf "" --class <class> <jar> ]


Cannot make the log4j portion above to work in Spark action - everything else is ok:


<action name="spark-7844">
  <spark xmlns="uri:oozie:spark-action:0.1">
    <spark-opts>--executor-memory 2G --files hdfs:/user/myao/config/ --conf &quot;; --conf &quot;; </spark-opts>
  <ok to="End"/>
  <error to="Kill"/>



Driver stderr:


Using properties file: null
Warning: Ignoring non-spark config property: ""
Warning: Ignoring non-spark config property: ""
Parsed arguments:
master yarn-cluster
deployMode cluster
executorMemory 2G
executorCores null
totalExecutorCores null
propertiesFile null
driverMemory null
driverCores null
driverExtraClassPath  ........

driverExtraLibraryPath /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop/lib/native
driverExtraJavaOptions null
supervise false
queue null
numExecutors null
files hdfs:/user/myao/config/
pyFiles null
archives null
mainClass ....
primaryResource ....
name ....
childArgs []

jar  .............
packages null
packagesExclusions null
repositories null
verbose true





What version are you using ? 

Can you send the workflow.xml generated by Hue ? 

Most likely it's fixed on the latest version.

; ;