Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Spark history server startup error

avatar

HDP-2.3.8.0-74

Spark 1.5.2.2.3

The start of "Spark history server" is successful in Ambari but very quickly it stops/fails. In the logs I see the following

16/01/01 16:42:36 INFO HistoryServer: History provider class: org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider
Exception in thread "main" java.lang.ClassNotFoundException: org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:191)
        at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:224)
        at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
16/01/01 16:42:36 INFO Utils: Shutdown hook called

The spark-defaults.conf is as follows

# Generated by Apache Ambari. Fri Jan  1 16:42:31 2016

spark.history.kerberos.keytab none
spark.history.kerberos.principal none
spark.history.provider org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider
spark.history.ui.port 18080
spark.yarn.containerLauncherMaxThreads 25
spark.yarn.driver.memoryOverhead 384
spark.yarn.executor.memoryOverhead 384
spark.yarn.historyServer.address host10.demo.com:18080  
spark.yarn.max.executor.failures 3
spark.yarn.preserve.staging.files false
spark.yarn.queue default
spark.yarn.scheduler.heartbeat.interval-ms 5000
spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryServicespark.yarn.submit.file.replication 3
1 ACCEPTED SOLUTION

avatar

Hmm, that's my code at fault there.

What it's saying is that the history server can't find the class org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider, which is what it is configured to use for histories.

The workaround is to switch to the classic filesystem history provider, which you can do

spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider

Deleting the line entirely should force the history server to revert to this.

I'd also look at deleting the line

spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService

If the timeline server integration is not on the classpath, the publisher is unlikely to be there too.

If that doesn't solve it, comment on this issue and I'll see what else we can do.

View solution in original post

5 REPLIES 5

avatar

Hmm, that's my code at fault there.

What it's saying is that the history server can't find the class org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider, which is what it is configured to use for histories.

The workaround is to switch to the classic filesystem history provider, which you can do

spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider

Deleting the line entirely should force the history server to revert to this.

I'd also look at deleting the line

spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService

If the timeline server integration is not on the classpath, the publisher is unlikely to be there too.

If that doesn't solve it, comment on this issue and I'll see what else we can do.

avatar

I'll try the FsHistoryProvider but that would mean the Spark history server is not integrated with Yarn ATS.

Are you saying that the ATS integration is not yet supported in the HDP stack version mentioned above or do you think this issue is specific to my setup ?

avatar

sorry, just catching up on this. ATS integration should be there, so I'm trying to work out why things don't work.

avatar

The same spark version 1.5.2.2.3 works fine with YarnHistoryProvider, at least the Spark history server started fine with HDP 2.3.6.0-3488 (note lower than the one in starting post of this thread)

avatar
@KRISHNANAND THOMMAND

Can you email me, thats stevel at hortonworks.com , the exact spark-assembly JAR you've got with this problem. I want to make sure that I really am looking at the one you are seeing this problem. Once I've got it I'll have a look inside the JAR to see what's up.

thanks