question Re: Adding Spark 2.2 in Archives of Support Questions (Read Only)

Adding Spark 2.2

avoda — Tue, 03 Oct 2017 19:20:06 GMT

Hi,

We have an Ambari cluster with HDFS, YARN and Map reduce 2 all at version 2.7.3

There is a built in option of adding Spark 1.6.3 or 2.1.1 to the cluster, but we are looking to install 2.2

Have anyone tried Spark 2.2 here yet, or is there an upgrade option we are not aware of?

Thanks!

Re: Adding Spark 2.2

ahadjidj — Tue, 03 Oct 2017 20:55:26 GMT

Hi @Aviram Voda

Spark 2.2 is not yet supported with HDP 2.6.x that's why it's not available through Ambari (Install/upgrade). It has to go through our QE to be added in further release.

Re: Adding Spark 2.2

avoda — Sun, 15 Oct 2017 14:47:06 GMT

Thanks Abdelkrim, do you know about any future timeline regarding the support of 2.2?

Re: Adding Spark 2.2

ahadjidj — Wed, 01 Nov 2017 05:19:02 GMT

Hi @Aviram Voda

Good news for your. Spark 2.2 is supported in HDP 2.6.3 annonced today : https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.3/bk_release-notes/content/comp_versions.html

Re: Adding Spark 2.2

yogen — Mon, 13 Nov 2017 23:21:48 GMT

@Abdelkrim Hadjidj

I tried this new version and when I load spark app using Yarn Cluster, I see the that the spark version is still: 2.1.1.2.6.2.0-205 and not 2.2...
What am I doing wrong?

Re: Adding Spark 2.2

shobhit — Sun, 20 May 2018 23:10:06 GMT

Hi @Abdelkrim Hadjidj

Is there any future timeline for support of Spark 2.3? The new structured streaming is critical to some of our features.