Cloudera plans on Spark 2.0.0

[#1]

(This doesn't sound related to this thread?)

[#2]

Sorry. I posted the question here because on 1st page it states: 'CDH is already effectively on 1.6.2'. And the bug should be fixed in 1.6.2.

I'll create a new topic for this.

[#3]

Hi

You strictly don't need to wait for Cloudera to release Spark 2.0.0. Since Spark can be run as a YARN application it is possible to run a Spark version other than the one that comes bundled with the Cloudera distribution. This requires no administrator privileges and no changes to the cluster configuration and can be done by any user who has permission to run a YARN  job on the cluster. A YARN application ships over all it’s dependencies over to the cluster for each invocation. You can run multiple Spark versions simultaneously on a YARN cluster. Each version of Spark is self contained in in the user workspace on the Edge node. Running a new Spark version will not affect any other jobs running on your cluster.

Essentially you download and extract spark-2.0.0 on the edge node, copy over your existing cluster configuration and hive-site.xml to the configuration directory and run spark-shell from the new location. This should work out of the box. There are a few optional configuration tweaks, you will find detailed instructions on how to do this if you google for them

Though note that jobs running on Spark 2.0.0 may not be supported by Cloudera till the official release.

Regards

Deenar

[#4]

@DeenarT: Question is not whether you can point to a spark distro rather it's natively available or not. On a side note, Spark has released 2.0.0. Not sure if you meant Cloudera's version.

[#5]

PS Spark 2 is now available as a 'beta' add-on for CDH 5.x, such that you can use both 2.0 and 1.6 on the same cluster.

http://www.cloudera.com/documentation/betas/spark2/latest/topics/spark2_release_notes.html

[#6]

Hello Sean,

any idea when we might get a Spark 2.0 GA for CDH?

kind regards

Geert

[#7]

It was released about a month ago: http://www.cloudera.com/downloads/spark2/2-0.html

[#8]

We are using Hive-on-Spark. Is there a documentation on how to use Spark2 for hive-on-spark?

Regards,

Vijay