We have an Ambari cluster with HDFS, YARN and Map reduce 2 all at version 2.7.3
There is a built in option of adding Spark 1.6.3 or 2.1.1 to the cluster, but we are looking to install 2.2
Have anyone tried Spark 2.2 here yet, or is there an upgrade option we are not aware of?
Hi @Aviram Voda
Spark 2.2 is not yet supported with HDP 2.6.x that's why it's not available through Ambari (Install/upgrade). It has to go through our QE to be added in further release.
View solution in original post
Thanks Abdelkrim, do you know about any future timeline regarding the support of 2.2?
Good news for your. Spark 2.2 is supported in HDP 2.6.3 annonced today : https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.3/bk_release-notes/content/comp_versions.html
I tried this new version and when I load spark app using Yarn Cluster, I see the that the spark version is still: 18.104.22.168.6.2.0-205 and not 2.2... What am I doing wrong?
Hi @Abdelkrim Hadjidj
Is there any future timeline for support of Spark 2.3? The new structured streaming is critical to some of our features.