Reply
Cloudera Employee
Posts: 481
Registered: ‎08-11-2014

Re: Cloudera plans on Spark 2.0.0

It's alluded to here: http://vision.cloudera.com/enhanced-streaming-and-machine-learning-with-apache-spark-2-0/ but it will be available as a 'beta' soon, though I have not heard anything more specific than that, though I also think it can't be long. The installed Spark in CDH 5.x must stay on Spark 1.x because Spark 2.x is not backwards-compatible, but it's possible to also have Spark 2.x installed and available alongside it. You can imagine that this will be the nature of how it's distributed, so not necessarily tied to the core CDH distribution.

Cloudera Employee
Posts: 481
Registered: ‎08-11-2014

Re: Cloudera plans on Spark 2.0.0

PS Spark 2 is now available as a 'beta' add-on for CDH 5.x, such that you can use both 2.0 and 1.6 on the same cluster.

 

http://www.cloudera.com/documentation/betas/spark2/latest/topics/spark2_release_notes.html

New Contributor
Posts: 5
Registered: ‎11-11-2015

Re: Cloudera plans on Spark 2.0.0

Hi,

 

We encountered the dead-lock error described in the issue: https://issues.apache.org/jira/browse/SPARK-13566. It should be fixed in 1.6.2. Can you tell me in which CDH-version is this fix included? We're using 'cdh5.9.0.p0.23'. Thx.

 

 

Cloudera Employee
Posts: 481
Registered: ‎08-11-2014

Re: Cloudera plans on Spark 2.0.0

(This doesn't sound related to this thread?)

New Contributor
Posts: 5
Registered: ‎11-11-2015

Re: Cloudera plans on Spark 2.0.0

Sorry. I posted the question here because on 1st page it states: 'CDH is already effectively on 1.6.2'. And the bug should be fixed in 1.6.2.

 

I'll create a new topic for this.

New Contributor
Posts: 6
Registered: ‎07-22-2014

Re: Cloudera plans on Spark 2.0.0

Hello Sean,

 

any idea when we might get a Spark 2.0 GA for CDH?

 

kind regards

Geert

Cloudera Employee
Posts: 481
Registered: ‎08-11-2014

Re: Cloudera plans on Spark 2.0.0

It was released about a month ago: http://www.cloudera.com/downloads/spark2/2-0.html

New Contributor
Posts: 2
Registered: ‎04-20-2016

Re: Cloudera plans on Spark 2.0.0

We are using Hive-on-Spark. Is there a documentation on how to use Spark2 for hive-on-spark?

 

Regards,

Vijay