Support Questions
Find answers, ask questions, and share your expertise

Add multiple options to spark.exeuctor.extraJavaOptions

Add multiple options to spark.exeuctor.extraJavaOptions

New Contributor

In HUE UI, using Spark action widget, how to set up multiple options?

 

Like

-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

 

This does not work in "Options List" box

--conf spark.exeuctor.extraJavaOptions="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"

2 REPLIES 2

Re: Add multiple options to spark.exeuctor.extraJavaOptions

Contributor

CDH 5.7.1 - Same issue, but for configuring app-specific log4j:

 

Working spark-submit command line: 

[--master yarn-cluster --files hdfs:/user/myao/config/log4j.properties --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties" --class <class> <jar> ]

 

Cannot make the log4j portion above to work in Spark action - everything else is ok:

 

<action name="spark-7844">
  <spark xmlns="uri:oozie:spark-action:0.1">
    <job-tracker>${jobTracker}</job-tracker>
    <name-node>${nameNode}</name-node>
    <configuration>
      <property>
        <name>spark.driver.extraJavaOptions</name>
        <value>-Dlog4j.configuration=log4j.properties</value>
      </property>
      <property>
        <name>spark.executor.extraJavaOptions</name>
        <value>-Dlog4j.configuration=log4j.properties</value>
      </property>
    </configuration>
    <master>yarn-cluster</master>
    <mode>cluster</mode>
    <name>...</name>
    <class>...</class>
    <jar>...</jar>
    <spark-opts>--executor-memory 2G --files hdfs:/user/myao/config/log4j.properties --conf &quot;spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties&quot; --conf &quot;spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties&quot; </spark-opts>
  </spark>
  <ok to="End"/>
  <error to="Kill"/>
</action>

 

 

Driver stderr:

 

Using properties file: null
Warning: Ignoring non-spark config property: "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties"
Warning: Ignoring non-spark config property: "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties"
Parsed arguments:
master yarn-cluster
deployMode cluster
executorMemory 2G
executorCores null
totalExecutorCores null
propertiesFile null
driverMemory null
driverCores null
driverExtraClassPath  ........

driverExtraLibraryPath /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop/lib/native
driverExtraJavaOptions null
supervise false
queue null
numExecutors null
files hdfs:/user/myao/config/log4j.properties
pyFiles null
archives null
mainClass ....
primaryResource ....
name ....
childArgs []

jar  .............
packages null
packagesExclusions null
repositories null
verbose true

 

Thanks!

 

Re: Add multiple options to spark.exeuctor.extraJavaOptions

Explorer

What version are you using ? 

Can you send the workflow.xml generated by Hue ? 

Most likely it's fixed on the latest version.