Hortonworks and Spark 2.0

[#1]

According to the Databricks people at the Spark Summit 2016 in New York, Spark 2.0 is coming out in April/May this year. This means a "jump" from 1.6.0 to 2.0.

Spark 1.5.2 is installed with the latest Hortonworks distribution. Spark 1.6.0 is in technical preview, Spark 2.0 is around the corner. Im wondering what plans Hortonworks has around this matter?

[#2 · Accepted solution]

Hi @marko, if I remember correctly Spark-1.6 TP for HDP-2.3.4 was announced 1 day after its GA. Therefore, you can expect Spark-2.0 TP with about the same "delay", and its inclusion in one of the coming versions of HDP soon after. Stay tuned!

[#3]

@vshukla

[#4]

No plans have been announced but we can guess around the same time as Apache stable release, so maybe when 2.0.1 lands.

[#5]

With HDP 2.4 release, we now support Spark 1.6.

[#6]

any guides to upgrade spark from 1.6.1 to 1.6.2 we are on the HDP 2.4.

[#7]

I'd like to upgrade my hortonworks 2.4.0.0-169 cluster to run Spark 2.0 now on GA to trial it, How to upgrade?