Is the version 2.0.0 of Spark supported in Hortonworks 2.4.x? Currently we have Spark 1.6 installed but we would need to update to 2.0.
Thanks a lot in advance.
Currently HDP 2.4.2 only support spark 1.6, therefore you won't see spark 2.0 in the repository. We don't recommend to install versions which are not tested which specific HDP release.
However if you are looking to test spark 2.0 on HDP then can you download the spark source code and build it with HDP dependencies.
View solution in original post
Spark 2.0 is not supported in 2.4.x.
Have a look here: https://hortonworks.com/blog/welcome-apache-spark-2-0/
Hope this helps.
Spark 2.0 is technical preview in HDP 2.5. You can try out 2.0 in Hortonworks cloud today or look for HDP 2.5 announcement to try it in your own env.