Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

CHD 5.7.1 + Spark 1.6 + spark.driver.extraJavaOptions issue

CHD 5.7.1 + Spark 1.6 + spark.driver.extraJavaOptions issue

New Contributor

Hi Team,

I am using CDH 5.7.1 with spark 1.6.0

I have a spark streaming application that read s from kafka & do some processing.

The issue is while starting the application in CLUSTER mode, i want to pass custom log4j.properies file to both driver & executor.

I have the below command :-

spark-submit \
--class xyx.search.spark.Boot \
--conf "spark.cores.max=6" \
--conf "spark.eventLog.enabled=true" \
--conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=file:/some/path/search-spark-service-log4j-Driver.properties" \
--conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/some/path/search-spark-service-log4j-Executor.properties" \

--deploy-mode "cluster" \
/some/path/search-spark-service-1.0.0.jar \
/some/path/conf/


But it gives the below exception :-

SPARK_JAVA_OPTS was detected (set to '-XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh ').
This is deprecated in Spark 1.0+.

Please instead use:
 - ./spark-submit with conf/spark-defaults.conf to set defaults for an application
 - ./spark-submit with --driver-java-options to set -X options for a driver
 - spark.executor.extraJavaOptions to set -X options for executors
 - SPARK_DAEMON_JAVA_OPTS to set java options for standalone daemons (master or worker)
       
2016-07-21 12:59:41 ERROR SparkContext:95 - Error initializing SparkContext.
org.apache.spark.SparkException: Found both spark.executor.extraJavaOptions and SPARK_JAVA_OPTS. Use only the former.
        at org.apache.spark.SparkConf$$anonfun$validateSettings$6$$anonfun$apply$5.apply(SparkConf.scala:470)
        at org.apache.spark.SparkConf$$anonfun$validateSettings$6$$anonfun$apply$5.apply(SparkConf.scala:468)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at org.apache.spark.SparkConf$$anonfun$validateSettings$6.apply(SparkConf.scala:468)
        at org.apache.spark.SparkConf$$anonfun$validateSettings$6.apply(SparkConf.scala:454)


Please note the same works with CDH 5.4 with spark 1.3.0.

Regards,
Sam

4 REPLIES 4

Re: CHD 5.7.1 + Spark 1.6 + spark.driver.extraJavaOptions issue

Master Collaborator

You've got SPARK_JAVA_OPTS set somewhere, as the error says, and you should remove it. It looks like you may have old or outdated spark-env.sh configuration somewhere in your cluster. I'd make sure every node is up to date.

Highlighted

Re: CHD 5.7.1 + Spark 1.6 + spark.driver.extraJavaOptions issue

New Contributor

Hi Sean,

 

I had checked the /opt/cloudera/parcels/CDH/lib/spark/conf/spark-env.sh in all nodes. None of then have SPARK_JAVA_OPTS exported.

 

This is a brand new cluster, initally i was working with 1.3.0 & the same command worked fine.

 

Is there any specific issue reported with CDH 5.7.1 + spark 1.6.0 reg extraJavaOptions

 

Thanks for your reply.

 

Regards,

Sam

Re: CHD 5.7.1 + Spark 1.6 + spark.driver.extraJavaOptions issue

Master Collaborator

I have 5.7.1 and do not observe this. Something is putting that config in your env. It looks like it might be something that CM sets for the Spark History Server, but you would not have that config in your environment by default. Something's cross-wired or sourcing the wrong thing, or old config is lying around.

Re: CHD 5.7.1 + Spark 1.6 + spark.driver.extraJavaOptions issue

Champion

@sam1988curious to know if you found any fix for this issue . I am landing on the same error and History server wont even start - its in down stage . any information is highly appreciable.

Don't have an account?
Coming from Hortonworks? Activate your account here