Spark 1.5.1 Tech Preview

[#1]

Team,

Congratulations. We just published Spark 1.5.1 TP (http://hortonworks.com/hadoop-tutorial/apache-spark-1-5-1-technical-preview-with-hdp-2-3/)

Next stop on this journey is Spark 1.5.1 GA with HDP.

Please take it out for a spin.

[#2]

Installed Spark 1.5.1 on Ambari installed 2.3.2.0-2950 cluster using below and then ran SparkPi job:

$ wget -nv  http://private-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.2.1-12/hdp.repo  -O /etc/yum.repos.d/HDP-TP.repo
$ yum install spark_2_3_2_1_12-master -y

$ su spark
$ echo $JAVA_HOME
/usr/java/default
$ cd /usr/hdp/2.3.2.1-12/spark
$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 lib/spark-examples*.jar 10

But am running into the same "ClassNotFoundException: org.apache.spark.deploy.yarn.history.YarnHistoryService" issue that others ran into. Have others been able to make this work successfully?

15/11/23 00:10:59 ERROR Utils: Uncaught exception in thread main
java.lang.NullPointerException
	at org.apache.spark.network.netty.NettyBlockTransferService.close(NettyBlockTransferService.scala:152)
	at org.apache.spark.storage.BlockManager.stop(BlockManager.scala:1228)
	at org.apache.spark.SparkEnv.stop(SparkEnv.scala:100)
	at org.apache.spark.SparkContext$$anonfun$stop$12.apply$mcV$sp(SparkContext.scala:1749)
	at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1185)
	at org.apache.spark.SparkContext.stop(SparkContext.scala:1748)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:593)
	at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:29)
	at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
15/11/23 00:10:59 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.history.YarnHistoryService
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:278)
	at org.apache.spark.util.Utils$.classForName(Utils.scala:173)
	at org.apache.spark.scheduler.cluster.YarnExtensionServices$$anonfun$start$5$$anonfun$apply$3.apply(YarnExtensionService.scala:106)
	at org.apache.spark.scheduler.cluster.YarnExtensionServices$$anonfun$start$5$$anonfun$apply$3.apply(YarnExtensionService.scala:105)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
	at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
	at org.apache.spark.scheduler.cluster.YarnExtensionServices$$anonfun$start$5.apply(YarnExtensionService.scala:105)
	at org.apache.spark.scheduler.cluster.YarnExtensionServices$$anonfun$start$5.apply(YarnExtensionService.scala:103)
	at scala.Option.map(Option.scala:145)
	at org.apache.spark.scheduler.cluster.YarnExtensionServices.start(YarnExtensionService.scala:103)
	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:64)
	at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:523)
	at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:29)
	at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
15/11/23 00:10:59 INFO DiskBlockManager: Shutdown hook called

[#3]

@Ali Bajwa same here with sandbox 2.3.2, when I try to run Spark SQL Thrift server.

@vshukla can you help? I think we also need to update our blog with instructions to solve this issue.

[#4]

There may have been an earlier instance of Spark where 1.5.1 TP was installed.

Check spark-default.conf it should NOT have any YARN ATS service related properties (enabled), since we don't have the ATS integration code ported to this TP.

Here is an example spark-default.conf that works.

# Default system properties included when running spark-submit.

# This is useful for setting default environmental settings.

# Example:

# spark.master spark://master:7077

# spark.eventLog.enabled true

# spark.eventLog.dir hdfs://namenode:8021/directory

# spark.serializer org.apache.spark.serializer.KryoSerializer

# spark.driver.memory 5g

# spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

#spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService

#spark.history.provider org.apache.spark.deploy.yarn.history.YarnHistoryProvider

## Make sure the host and port match the node where your YARN history server is running

#spark.yarn.historyServer.address localhost:18080

spark.driver.extraJavaOptions -Dhdp.version=2.3.2.1-12

spark.yarn.am.extraJavaOptions -Dhdp.version=2.3.2.1-12

[#5]

Vinay -we do have the ATS integration in this TP, we just omitted something to bridge from the binding classes used in Ambari since the 1.2/1.3 previews to their current names. That's now been fixed, and there is the workaround listed above to address it directly.

[#6]

Can you please elaborate on the this workaround? I would be nice to have ATS integration for Spark 1.5.1.

[#7]

You can install the latest HDP 2.3.4 using Ambari 2.2.0.0: it comes with Spark 1.5.2 and its integrated with ATS

[#8 · Accepted solution]

Thank you @vshukla

Removing following properties from /etc/spark/conf/spark-defaults.conf solved the issue:

#spark.history.provider org.apache.spark.deploy.yarn.history.YarnHistoryProvider
#spark.history.ui.port 18080
#spark.yarn.historyServer.address sandbox.hortonworks.com:18080
#spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService

PS: I installed spark in sandbox 2.3.2 and followed your instructions (http://hortonworks.com/hadoop-tutorial/apache-spark-1-5-1-technical-preview-with-hdp-2-3/). It would be nice if you could update spark 1.5.1 tutorial as others users are having same issue @Ali Bajwa.

[#9]

Thanks that worked! Yes we should def update the TP instructions because multiple folks on forum had this issue too:

In case anyone else runs into the same, here is a script to automate it

sed -i /spark.history.provider/s/^/#/ /usr/hdp/2.3.2.1-12/spark/conf/spark-defaults.conf
sed -i /spark.history.ui.port/s/^/#/ /usr/hdp/2.3.2.1-12/spark/conf/spark-defaults.conf
sed -i /spark.yarn.historyServer.address/s/^/#/ /usr/hdp/2.3.2.1-12/spark/conf/spark-defaults.conf
sed -i /spark.yarn.services/s/^/#/ /usr/hdp/2.3.2.1-12/spark/conf/spark-defaults.conf