Reply
Explorer
Posts: 6
Registered: ‎01-27-2017

Spark job fails from Oozie workflow, runs from spark-submit

Hi all!

 

We're using CDH 5.10.0. We create jar and run it via spark-submit where master is yarn-cluster and mode is cluster.

 

We get following exception:

Launcher exception: org.apache.spark.SecurityManager$.$lessinit$greater$default$2()Lscala/Option;
java.lang.NoSuchMethodError: org.apache.spark.SecurityManager$.$lessinit$greater$default$2()Lscala/Option;
	at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:878)
	at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:143)
	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
	at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:157)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:542)
	at de.gameduell.bits.spark.test.AvroReader.main(AvroReader.java:13)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
	at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:338)
	at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:257)
	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:60)
	at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:78)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:232)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Now, this seems especially weird as org.apache.spark.SecurityManager class is not present in Spark 1.6.0. The last version that has it is Spark 1.3.0. We're not using anything custom here, everything is from CDH 5.10 package. Any pointer where to look for rogue libraries would be appreciated.

Posts: 642
Topics: 3
Kudos: 103
Solutions: 67
Registered: ‎08-16-2016

Re: Spark job fails from Oozie workflow, runs from spark-submit

My first guess would be that the shared library isn't configured or the needed Spark library is not in it.
Highlighted
Cloudera Employee
Posts: 3
Registered: ‎06-21-2016

Re: Spark job fails from Oozie workflow, runs from spark-submit

Hi akarac,

Passing -verbose:class JVM option to the Spark driver would help you see what classes are loaded from where. I suspect an unintended library version (e.g. a transitive dependency or something from a shaded jar) was pulled in in your case.  

In your workflow, add the following to  spark-opts

--conf spark.driver.extraJavaOptions="-verbose:class"

If you rerun the workflow, you should see in the Oozie Launcher's log what classes are loaded.

[Loaded org.apache.spark.SecurityManager from  ... ]


To list your sharelib, you can run:

oozie admin -shareliblist spark

 

Announcements