Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark job fails from Oozie workflow, runs from spark-submit

Highlighted

Spark job fails from Oozie workflow, runs from spark-submit

Explorer

Hi all!

 

We're using CDH 5.10.0. We create jar and run it via spark-submit where master is yarn-cluster and mode is cluster.

 

We get following exception:

Launcher exception: org.apache.spark.SecurityManager$.$lessinit$greater$default$2()Lscala/Option;
java.lang.NoSuchMethodError: org.apache.spark.SecurityManager$.$lessinit$greater$default$2()Lscala/Option;
	at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:878)
	at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:143)
	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
	at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:157)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:542)
	at de.gameduell.bits.spark.test.AvroReader.main(AvroReader.java:13)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
	at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:338)
	at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:257)
	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:60)
	at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:78)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:232)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Now, this seems especially weird as org.apache.spark.SecurityManager class is not present in Spark 1.6.0. The last version that has it is Spark 1.3.0. We're not using anything custom here, everything is from CDH 5.10 package. Any pointer where to look for rogue libraries would be appreciated.

2 REPLIES 2

Re: Spark job fails from Oozie workflow, runs from spark-submit

Champion
My first guess would be that the shared library isn't configured or the needed Spark library is not in it.

Re: Spark job fails from Oozie workflow, runs from spark-submit

Cloudera Employee

Hi akarac,

Passing -verbose:class JVM option to the Spark driver would help you see what classes are loaded from where. I suspect an unintended library version (e.g. a transitive dependency or something from a shaded jar) was pulled in in your case.  

In your workflow, add the following to  spark-opts

--conf spark.driver.extraJavaOptions="-verbose:class"

If you rerun the workflow, you should see in the Oozie Launcher's log what classes are loaded.

[Loaded org.apache.spark.SecurityManager from  ... ]


To list your sharelib, you can run:

oozie admin -shareliblist spark