Reply
Contributor
Posts: 30
Registered: ‎04-07-2016

spark 2.0 with spark1.6

[ Edited ]

Hi,

I just saw that blog post about spark 2.0 beta : http://blog.cloudera.com/blog/2016/09/apache-spark-2-0-beta-now-available-for-cdh/

And I have a quick question, once spark 2.0 is installed on the cluster, how do I pick If i want my job to go to spark2.0 or 1.6?

 

thanks

Cloudera Employee
Posts: 7
Registered: ‎07-20-2014

Re: spark 2.0 with spark1.6

Hi Maurin,

You use spark2-submit for Spark 2.0 and spark-submit for CDH Spark (1.6).

More details can be found in our documentation:
http://www.cloudera.com/documentation/betas/spark2/latest/topics/spark_running_apps.html

Best,
Bjorn
Explorer
Posts: 12
Registered: ‎05-25-2016

Re: spark 2.0 with spark1.6

I installed Spark2 service as mentioned in the installation, can you please let me know why it is invoking Spark 1.6 when I am executing spark2-submit.

 

Here is the command:

 spark2-shell --master local[4]

 

Here are the logs for the same.

 

Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.6.0
/_/

Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_79)
Type in expressions to have them evaluated.
Type :help for more information.
16/12/01 03:15:31 INFO spark.SparkContext: Running Spark version 1.6.0
16/12/01 03:15:32 INFO util.ShutdownHookManager: Shutdown hook called
16/12/01 03:15:32 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-11270a13-582a-44d5-b639-9c6c91d59239/repl-7a4740ff-ebec-4767-9a98-026320a84c6f
16/12/01 03:15:32 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-11270a13-582a-44d5-b639-9c6c91d59239

New Contributor
Posts: 7
Registered: ‎08-26-2015

Re: spark 2.0 with spark1.6

Just installed the Spark2 parcel to CDH 5.9, and when starting the spark-shell I'm getting, class not found:

 

> spark2-shell --master local[4]

 

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:118)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:118)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:118)
at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:104)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:117)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more

Cloudera Employee
Posts: 461
Registered: ‎08-11-2014

Re: spark 2.0 with spark1.6

I can't reproduce either of the last two problems, but the second one may be because you're trying to use a local master. You need to use YARN.

New Contributor
Posts: 7
Registered: ‎08-26-2015

Re: spark 2.0 with spark1.6

Same problem:

 

>spark2-shell --master yarn --driver-cores 20 --num-executors 4 --executor-cores 20 --driver-memory 20g --executor-memory 20g
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:118)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:118)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:118)
at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:104)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:117)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more

 

spark-shell from 1.6 starts without any problem...

Cloudera Employee
Posts: 461
Registered: ‎08-11-2014

Re: spark 2.0 with spark1.6

I can run spark2-shell with local master or YARN. These errors indicate some kind of classpath problem. Is it possible you have customized your installation somehow, or have stray configuration in effect? env variables? I have a vanilla 5.9 cluster with the Spark 2 CSD and don't see this.

Cloudera Employee
Posts: 94
Registered: ‎05-10-2016

Re: spark 2.0 with spark1.6

Along with Sean's thoughts, also make sure you are executing spark2-shell from a Spark2 gateway[1] and have deployed client configurations.  This process is similar for both spark and spark2, but you do need to ensure it is completed for both services within the cluster in cloudera manager.

 

1.  https://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_managing_roles.html#xd_583c10b...

Highlighted
New Contributor
Posts: 7
Registered: ‎08-26-2015

Re: spark 2.0 with spark1.6

Thank you very much - assigning of roles solved the problem...

Announcements