Created on 04-06-2016 11:33 AM - edited 09-16-2022 03:12 AM
In HUE UI, using Spark action widget, how to set up multiple options?
Like
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
This does not work in "Options List" box
--conf spark.exeuctor.extraJavaOptions="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
Created on 09-15-2016 12:26 PM - edited 09-15-2016 12:31 PM
CDH 5.7.1 - Same issue, but for configuring app-specific log4j:
Working spark-submit command line:
[--master yarn-cluster --files hdfs:/user/myao/config/log4j.properties --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties" --class <class> <jar> ]
Cannot make the log4j portion above to work in Spark action - everything else is ok:
<action name="spark-7844">
<spark xmlns="uri:oozie:spark-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>spark.driver.extraJavaOptions</name>
<value>-Dlog4j.configuration=log4j.properties</value>
</property>
<property>
<name>spark.executor.extraJavaOptions</name>
<value>-Dlog4j.configuration=log4j.properties</value>
</property>
</configuration>
<master>yarn-cluster</master>
<mode>cluster</mode>
<name>...</name>
<class>...</class>
<jar>...</jar>
<spark-opts>--executor-memory 2G --files hdfs:/user/myao/config/log4j.properties --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties" </spark-opts>
</spark>
<ok to="End"/>
<error to="Kill"/>
</action>
Driver stderr:
Using properties file: null
Warning: Ignoring non-spark config property: "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties"
Warning: Ignoring non-spark config property: "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties"
Parsed arguments:
master yarn-cluster
deployMode cluster
executorMemory 2G
executorCores null
totalExecutorCores null
propertiesFile null
driverMemory null
driverCores null
driverExtraClassPath ........
driverExtraLibraryPath /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop/lib/native
driverExtraJavaOptions null
supervise false
queue null
numExecutors null
files hdfs:/user/myao/config/log4j.properties
pyFiles null
archives null
mainClass ....
primaryResource ....
name ....
childArgs []
jar .............
packages null
packagesExclusions null
repositories null
verbose true
Thanks!
Created 10-03-2016 05:35 PM
What version are you using ?
Can you send the workflow.xml generated by Hue ?
Most likely it's fixed on the latest version.