question Re: Spark version supported in Hortonworks 2.4.x in Archives of Support Questions (Read Only)

Spark version supported in Hortonworks 2.4.x

jaimecarodrigue — Wed, 03 Aug 2016 17:03:00 GMT

Hi,

Is the version 2.0.0 of Spark supported in Hortonworks 2.4.x? Currently we have Spark 1.6 installed but we would need to update to 2.0.

Thanks a lot in advance.

Re: Spark version supported in Hortonworks 2.4.x

jyadav — Wed, 03 Aug 2016 17:09:34 GMT

Hi @Jaime

Currently HDP 2.4.2 only support spark 1.6, therefore you won't see spark 2.0 in the repository. We don't recommend to install versions which are not tested which specific HDP release.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_HDP_RelNotes/content/ch_relnotes_v242.html

However if you are looking to test spark 2.0 on HDP then can you download the spark source code and build it with HDP dependencies.

Re: Spark version supported in Hortonworks 2.4.x

pvillard — Wed, 03 Aug 2016 17:10:58 GMT

Hi @Jaime,

Spark 2.0 is not supported in 2.4.x.

Have a look here: https://hortonworks.com/blog/welcome-apache-spark-2-0/

Hope this helps.

Re: Spark version supported in Hortonworks 2.4.x

vshukla — Wed, 03 Aug 2016 23:31:01 GMT

Spark 2.0 is technical preview in HDP 2.5. You can try out 2.0 in Hortonworks cloud today or look for HDP 2.5 announcement to try it in your own env.

Re: Spark version supported in Hortonworks 2.4.x

jaimecarodrigue — Thu, 04 Aug 2016 13:27:32 GMT

Thanks @vshukla.