Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Add multiple options to spark.exeuctor.extraJavaOptions

avatar
New Contributor

In HUE UI, using Spark action widget, how to set up multiple options?

 

Like

-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

 

This does not work in "Options List" box

--conf spark.exeuctor.extraJavaOptions="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"

2 REPLIES 2

avatar
Expert Contributor

CDH 5.7.1 - Same issue, but for configuring app-specific log4j:

 

Working spark-submit command line: 

[--master yarn-cluster --files hdfs:/user/myao/config/log4j.properties --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties" --class <class> <jar> ]

 

Cannot make the log4j portion above to work in Spark action - everything else is ok:

 

<action name="spark-7844">
  <spark xmlns="uri:oozie:spark-action:0.1">
    <job-tracker>${jobTracker}</job-tracker>
    <name-node>${nameNode}</name-node>
    <configuration>
      <property>
        <name>spark.driver.extraJavaOptions</name>
        <value>-Dlog4j.configuration=log4j.properties</value>
      </property>
      <property>
        <name>spark.executor.extraJavaOptions</name>
        <value>-Dlog4j.configuration=log4j.properties</value>
      </property>
    </configuration>
    <master>yarn-cluster</master>
    <mode>cluster</mode>
    <name>...</name>
    <class>...</class>
    <jar>...</jar>
    <spark-opts>--executor-memory 2G --files hdfs:/user/myao/config/log4j.properties --conf &quot;spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties&quot; --conf &quot;spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties&quot; </spark-opts>
  </spark>
  <ok to="End"/>
  <error to="Kill"/>
</action>

 

 

Driver stderr:

 

Using properties file: null
Warning: Ignoring non-spark config property: "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties"
Warning: Ignoring non-spark config property: "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties"
Parsed arguments:
master yarn-cluster
deployMode cluster
executorMemory 2G
executorCores null
totalExecutorCores null
propertiesFile null
driverMemory null
driverCores null
driverExtraClassPath  ........

driverExtraLibraryPath /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop/lib/native
driverExtraJavaOptions null
supervise false
queue null
numExecutors null
files hdfs:/user/myao/config/log4j.properties
pyFiles null
archives null
mainClass ....
primaryResource ....
name ....
childArgs []

jar  .............
packages null
packagesExclusions null
repositories null
verbose true

 

Thanks!

 

avatar
Explorer

What version are you using ? 

Can you send the workflow.xml generated by Hue ? 

Most likely it's fixed on the latest version.