Reply
New Contributor
Posts: 3
Registered: ‎10-29-2016

spark-submit fails after setting userClassPathFirst to true

After i set spark.driver.userClassPathFirst=true, my spark-submit --master yarn-client fails with below error & it works fine if i remove userClassPathFirst setting. I need to add this setting to avoid class conflicts in some other job so trying to make it this setting work in simple job first & later try with job with class conflicts. 

 

From quick search looks like this error occurs when driver cannot find yarn & hadoop related config, so exported SPARK_CONF_DIR & HADOOP_CONF_DIR and also added config files in --jar option but still get the same error.

 

Any ideas on how to fix this?

 

org.apache.spark.SparkException: Unable to load YARN support

        at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:399)

        at org.apache.spark.deploy.SparkHadoopUtil$.yarn$lzycompute(SparkHadoopUtil.scala:394)

        at org.apache.spark.deploy.SparkHadoopUtil$.yarn(SparkHadoopUtil.scala:394)

        at org.apache.spark.deploy.SparkHadoopUtil$.get(SparkHadoopUtil.scala:411)

        at org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.scala:2119)

        at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:105)

        at org.apache.spark.SparkEnv$.create(SparkEnv.scala:365)

        at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:193)

        at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:289)

        at org.apache.spark.SparkContext.<init>(SparkContext.scala:462)

        at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59)

        at com.citi.ripcurl.timeseriesbatch.BatchContext.<init>(BatchContext.java:27)

        at com.citi.ripcurl.timeseriesbatch.example.EqDataQualityExample.runReportQuery(EqDataQualityExample.java:28)

        at com.citi.ripcurl.timeseriesbatch.example.EqDataQualityExample.main(EqDataQualityExample.java:70)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:497)

        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)

        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)

        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)

        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class org.apache.hadoop.security.ShellBasedUnixGroupsMapping not org.apache.hadoop.security.GroupMappingServiceProvider

Cloudera Employee
Posts: 94
Registered: ‎05-10-2016

Re: spark-submit fails after setting userClassPathFirst to true

The issues stems from dependencies without your user classpath being incompatible with spark libraries.  You will need to shade or remove any spark dependencies and spark transitive dependencies from your user classpath first or you may get some version conflicts like this.

New Contributor
Posts: 1
Registered: ‎08-31-2017

Re: spark-submit fails after setting userClassPathFirst to true

Hello, were you able to resolve this problem? I am facing the same issue now.

Announcements