Adding Spark 2.2

[#1]

Hi,

We have an Ambari cluster with HDFS, YARN and Map reduce 2 all at version 2.7.3

There is a built in option of adding Spark 1.6.3 or 2.1.1 to the cluster, but we are looking to install 2.2

Have anyone tried Spark 2.2 here yet, or is there an upgrade option we are not aware of?

Thanks!

[#2 · Accepted solution]

Hi @Aviram Voda

Spark 2.2 is not yet supported with HDP 2.6.x that's why it's not available through Ambari (Install/upgrade). It has to go through our QE to be added in further release.

[#3]

Thanks Abdelkrim, do you know about any future timeline regarding the support of 2.2?

[#4]

Hi @Aviram Voda

Good news for your. Spark 2.2 is supported in HDP 2.6.3 annonced today : https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.3/bk_release-notes/content/comp_versions.html

[#5]

@Abdelkrim Hadjidj

I tried this new version and when I load spark app using Yarn Cluster, I see the that the spark version is still: 2.1.1.2.6.2.0-205 and not 2.2...
What am I doing wrong?

[#6]

Hi @Abdelkrim Hadjidj

Is there any future timeline for support of Spark 2.3? The new structured streaming is critical to some of our features.