Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

spark-submit fails after setting userClassPathFirst to true

Highlighted

spark-submit fails after setting userClassPathFirst to true

New Contributor

After i set spark.driver.userClassPathFirst=true, my spark-submit --master yarn-client fails with below error & it works fine if i remove userClassPathFirst setting. I need to add this setting to avoid class conflicts in some other job so trying to make it this setting work in simple job first & later try with job with class conflicts. 

 

From quick search looks like this error occurs when driver cannot find yarn & hadoop related config, so exported SPARK_CONF_DIR & HADOOP_CONF_DIR and also added config files in --jar option but still get the same error.

 

Any ideas on how to fix this?

 

org.apache.spark.SparkException: Unable to load YARN support

        at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:399)

        at org.apache.spark.deploy.SparkHadoopUtil$.yarn$lzycompute(SparkHadoopUtil.scala:394)

        at org.apache.spark.deploy.SparkHadoopUtil$.yarn(SparkHadoopUtil.scala:394)

        at org.apache.spark.deploy.SparkHadoopUtil$.get(SparkHadoopUtil.scala:411)

        at org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.scala:2119)

        at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:105)

        at org.apache.spark.SparkEnv$.create(SparkEnv.scala:365)

        at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:193)

        at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:289)

        at org.apache.spark.SparkContext.<init>(SparkContext.scala:462)

        at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59)

        at com.citi.ripcurl.timeseriesbatch.BatchContext.<init>(BatchContext.java:27)

        at com.citi.ripcurl.timeseriesbatch.example.EqDataQualityExample.runReportQuery(EqDataQualityExample.java:28)

        at com.citi.ripcurl.timeseriesbatch.example.EqDataQualityExample.main(EqDataQualityExample.java:70)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:497)

        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)

        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)

        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)

        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class org.apache.hadoop.security.ShellBasedUnixGroupsMapping not org.apache.hadoop.security.GroupMappingServiceProvider

2 REPLIES 2

Re: spark-submit fails after setting userClassPathFirst to true

Expert Contributor

The issues stems from dependencies without your user classpath being incompatible with spark libraries.  You will need to shade or remove any spark dependencies and spark transitive dependencies from your user classpath first or you may get some version conflicts like this.

Re: spark-submit fails after setting userClassPathFirst to true

New Contributor

Hello, were you able to resolve this problem? I am facing the same issue now.