Upgrade Spark 2.1 (HDS-2.6.2.0) to Spark 2.2

[#1]

Thanks in advance on this - I am running Ambari and have deployed HDP-2.6.2.0 on a 12-node cluster (one name node, one secondary name-node and 10 data nodes). Originally when I did this I deployed Spark 2.1.1, HDFS 2.7.3 and other dependencies. One of our data scientists stated that he wants to use "computeSVD", but it is only available via the Python API in Spark 2.2. I'd like to upgrade Spark in place, but not sure if I need to upgrade other things, if I can do this via Ambari or what. Is there a process for doing this? Is Spark 2.2 provided in HDP at all yet? Thanks!

[#2]

Hi @Charles Bradbury

Spark 2.2 is not available in HDP so you won't be able to upgrade to this version using Ambari.

You can manually install Spark 2.2 on the cluster but it wasn't tested and is not certified/supported by Hortonworks yet.

Thanks

[#3]

Since Spark 2.2 is not provided by HDP (yet), and we are trying to use "computeSVD", is there an alternative for the same functionality under Spark 2.1? Basically the code we are using to compute the singular decomposition value of a matrix of message identifiers needs functionality from that "computeSVD". Its provided in the Scala API of Spark 2.1.1, but not in the Python API. Is there something else I can use for this?

[#4 · Accepted solution]

Hi @Charles Bradbury

For your information, Spark 2.2 is supported in HDP 2.6.3 annonced today : https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.3/bk_release-notes/content/comp_versions.html