08-23-2016 05:30 AM
It's alluded to here: http://vision.cloudera.com/enhanced-streaming-and-machine-learning-with-apache-spark-2-0/ but it will be available as a 'beta' soon, though I have not heard anything more specific than that, though I also think it can't be long. The installed Spark in CDH 5.x must stay on Spark 1.x because Spark 2.x is not backwards-compatible, but it's possible to also have Spark 2.x installed and available alongside it. You can imagine that this will be the nature of how it's distributed, so not necessarily tied to the core CDH distribution.
09-30-2016 03:57 PM
PS Spark 2 is now available as a 'beta' add-on for CDH 5.x, such that you can use both 2.0 and 1.6 on the same cluster.
11-16-2016 07:51 AM
We encountered the dead-lock error described in the issue: https://issues.apache.org/jira/browse/SPARK-13566. It should be fixed in 1.6.2. Can you tell me in which CDH-version is this fix included? We're using 'cdh5.9.0.p0.23'. Thx.
11-16-2016 08:12 AM
Sorry. I posted the question here because on 1st page it states: 'CDH is already effectively on 1.6.2'. And the bug should be fixed in 1.6.2.
I'll create a new topic for this.