Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession when running Spark 2.0+ example

avatar
Explorer

While running the Spark 2.0 example in this tutorial in the Sandbox, I am getting this error. I have followed the steps given and I am not able to find the issue. Has any faced this issue?

----------------------------------------------------------------------------------------------------------------------------------------

[root@sandbox ~]# export SPARK_HOME=/usr/hdp/current/spark-client

[root@sandbox ~]# export SPARK_MAJOR_VERSION=2

[root@sandbox ~]# cd /usr/hdp/current/spark2-client

[root@sandbox spark2-client]# su spark

[spark@sandbox spark2-client]$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 examples/jars/spark-examples*.jar 10 Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession$ at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:28) at org.apache.spark.examples.SparkPi.main(SparkPi.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession$ at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 11 more

2 REPLIES 2

avatar

Make sure you set the correct spark home which points to spark2-client.

export SPARK_HOME=/usr/hdp/current/spark2-client

You must be pointing it to export SPARK_HOME=/usr/hdp/current/spark-client

avatar
Master Guru

SPARK_HOME must be in your path

Check your classpath.

Make sure Java is working.