Spark history server startup error

[#1]

HDP-2.3.8.0-74

Spark 1.5.2.2.3

The start of "Spark history server" is successful in Ambari but very quickly it stops/fails. In the logs I see the following

16/01/01 16:42:36 INFO HistoryServer: History provider class: org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider

Exception in thread "main" java.lang.ClassNotFoundException: org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider

        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

        at java.security.AccessController.doPrivileged(Native Method)

        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:191)

        at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:224)

        at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)

16/01/01 16:42:36 INFO Utils: Shutdown hook called

The spark-defaults.conf is as follows

# Generated by Apache Ambari. Fri Jan  1 16:42:31 2016

spark.history.kerberos.keytab none
spark.history.kerberos.principal none
spark.history.provider org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider
spark.history.ui.port 18080
spark.yarn.containerLauncherMaxThreads 25
spark.yarn.driver.memoryOverhead 384
spark.yarn.executor.memoryOverhead 384
spark.yarn.historyServer.address host10.demo.com:18080  
spark.yarn.max.executor.failures 3
spark.yarn.preserve.staging.files false
spark.yarn.queue default
spark.yarn.scheduler.heartbeat.interval-ms 5000
spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryServicespark.yarn.submit.file.replication 3

[#2 · Accepted solution]

Hmm, that's my code at fault there.

What it's saying is that the history server can't find the class org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider, which is what it is configured to use for histories.

The workaround is to switch to the classic filesystem history provider, which you can do

spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider

Deleting the line entirely should force the history server to revert to this.

I'd also look at deleting the line

spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService

If the timeline server integration is not on the classpath, the publisher is unlikely to be there too.

If that doesn't solve it, comment on this issue and I'll see what else we can do.

[#3]

I'll try the FsHistoryProvider but that would mean the Spark history server is not integrated with Yarn ATS.

Are you saying that the ATS integration is not yet supported in the HDP stack version mentioned above or do you think this issue is specific to my setup ?

[#4]

sorry, just catching up on this. ATS integration should be there, so I'm trying to work out why things don't work.

[#5]

The same spark version 1.5.2.2.3 works fine with YarnHistoryProvider, at least the Spark history server started fine with HDP 2.3.6.0-3488 (note lower than the one in starting post of this thread)

[#6]

@KRISHNANAND THOMMAND

Can you email me, thats stevel at hortonworks.com , the exact spark-assembly JAR you've got with this problem. I want to make sure that I really am looking at the one you are seeing this problem. Once I've got it I'll have a look inside the JAR to see what's up.

thanks