Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession when running Spark 2.0+ example

java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession when running Spark 2.0+ example

Explorer

While running the Spark 2.0 example in this tutorial in the Sandbox, I am getting this error. I have followed the steps given and I am not able to find the issue. Has any faced this issue?

----------------------------------------------------------------------------------------------------------------------------------------

[root@sandbox ~]# export SPARK_HOME=/usr/hdp/current/spark-client

[root@sandbox ~]# export SPARK_MAJOR_VERSION=2

[root@sandbox ~]# cd /usr/hdp/current/spark2-client

[root@sandbox spark2-client]# su spark

[spark@sandbox spark2-client]$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 examples/jars/spark-examples*.jar 10 Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession$ at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:28) at org.apache.spark.examples.SparkPi.main(SparkPi.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession$ at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 11 more

2 REPLIES 2
Highlighted

Re: java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession when running Spark 2.0+ example

Make sure you set the correct spark home which points to spark2-client.

export SPARK_HOME=/usr/hdp/current/spark2-client

You must be pointing it to export SPARK_HOME=/usr/hdp/current/spark-client

Highlighted

Re: java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession when running Spark 2.0+ example

Super Guru

SPARK_HOME must be in your path

Check your classpath.

Make sure Java is working.

Don't have an account?
Coming from Hortonworks? Activate your account here