Spark version supported in Hortonworks 2.4.x

[#1]

Hi,

Is the version 2.0.0 of Spark supported in Hortonworks 2.4.x? Currently we have Spark 1.6 installed but we would need to update to 2.0.

Thanks a lot in advance.

[#2 · Accepted solution]

Hi @Jaime

Currently HDP 2.4.2 only support spark 1.6, therefore you won't see spark 2.0 in the repository. We don't recommend to install versions which are not tested which specific HDP release.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_HDP_RelNotes/content/ch_relnotes_v242.ht...

However if you are looking to test spark 2.0 on HDP then can you download the spark source code and build it with HDP dependencies.

[#3]

Hi @Jaime,

Spark 2.0 is not supported in 2.4.x.

Have a look here: https://hortonworks.com/blog/welcome-apache-spark-2-0/

Hope this helps.

[#4]

Spark 2.0 is technical preview in HDP 2.5. You can try out 2.0 in Hortonworks cloud today or look for HDP 2.5 announcement to try it in your own env.

[#5]

Thanks @vshukla.