Reply
Explorer
Posts: 24
Registered: ‎08-15-2016
Accepted Solution

Unable to run Spark on Cloudera Manager Single User Mode with JDK1.8

I have installed Cloudera Manager using the Single User Mode with Installation Path A, using JDK 1.8. The installation went fine, though when I try to run "spark-shell", I get the error "java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream" as  below:

 

 

org.apache.spark.launcher.app.Driver                         I Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
org.apache.spark.launcher.app.Driver                         I      at org.apache.spark.deploy.SparkSubmitArguments.handle(SparkSubmitArguments.scala:394)
org.apache.spark.launcher.app.Driver                         I      at org.apache.spark.launcher.SparkSubmitOptionParser.parse(SparkSubmitOptionParser.java:163)
org.apache.spark.launcher.app.Driver                         I      at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:97)
org.apache.spark.launcher.app.Driver                         I      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:114)
org.apache.spark.launcher.app.Driver                         I      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
org.apache.spark.launcher.app.Driver                         I Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
org.apache.spark.launcher.app.Driver                         I      at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
org.apache.spark.launcher.app.Driver                         I      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
org.apache.spark.launcher.app.Driver                         I      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
org.apache.spark.launcher.app.Driver                         I      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
org.apache.spark.launcher.app.Driver                         I      ... 5 more

I tried sourcing /etc/conf/spark/spark-env.sh and /etc/hadoop/conf/hadoop-env.sh manually, but did not help. What can I do to resolve this issue? Since I have installed Cloudera in the Single User Mode, is there anything additional that I need to do here?

 

Expert Contributor
Posts: 82
Registered: ‎02-24-2016

Re: Unable to run Spark on Cloudera Manager Single User Mode with JDK1.8

Explorer
Posts: 24
Registered: ‎08-15-2016

Re: Unable to run Spark on Cloudera Manager Single User Mode with JDK1.8

[ Edited ]

Hadoop is installed. A simple "hadoop version" yields this:

 

Hadoop 2.6.0-cdh5.8.0
Subversion http://github.com/cloudera/hadoop -r 042da8b868a212c843bcbf3594519dd26e816e79
Compiled by jenkins on 2016-07-12T23:02Z
Compiled with protoc 2.5.0
From source with checksum 2b6c31ecc19f118d6e1c822175716b5
This command was run using /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/jars/hadoop-common-2.6.0-cdh5.8.0.jar

Also, in the last lines it clearly mentions the hadoop-common.jar which is also the jar containg the error I am getting mentioned in the question. 

 

Any other suggestions?

Highlighted
Posts: 1,043
Topics: 1
Kudos: 262
Solutions: 130
Registered: ‎04-22-2014

Re: Unable to run Spark on Cloudera Manager Single User Mode with JDK1.8

HI,

 

It is likely that the Spark client configuration is not found on this host.

Try doing the following:

 

  1. in Cloudera Manager, go to Spark --> Instances --> Add Role Instances.  Add a Gateway to this host
  2. In Cloudera Manager, go to the Spark service and click the "Actions" menu; choose "Deploy Client Configuration"
  3. Test spark-shell

Based on the error, I think that spark-shell is not able to find an updated version of the client configuration.  If you already have a Gateway, then I'd deploy client configuration and see if that helps.

 

Regards,

 

Ben

Announcements