Spark version different to expected in HDP 2.3.2

[#1]

We have installed a fresh HDP 2.3.2 cluster and was expecting Spark 1.4.1 (as per build metadata, release notes etc.) but got Spark 1.5.2

Is this expected?

The repo was changed to 2.3.2 and only the redhat7 one was selected during installation..see screenshot.

thanks for your help, Alan

[#2 · Accepted solution]

@Alan McShane

you can confirm by going to the host and checking in /etc/yum.repos.d/ in HDP.repo file. Please double check whether repo is pointing to 2.3.4 and not 2.3.2. Spark 1.5.2 ships with HDP 2.3.4, Spark 1.4.1 ships with HDP 2.3.2. We just shipped Spark 1.6 with HDP 2.4. Why would you not want Spark 1.5.2 as that has newer features and stabilization fixes on top of 1.4.1.

[#3]

@Artem Ervits

Thanks for the quick reply 🙂 The HDP.repo is saying it is definitely 2.3.2 okay.

Yum is also showing the packages are Spark 1.4.1 as expected.

spark.noarch  1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark-master.noarch  1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark-python.noarch  1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark-worker.noarch  1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark_2_3_2_0_2950.noarch  1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark_2_3_2_0_2950-master.noarch 1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark_2_3_2_0_2950-python.noarch 1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark_2_3_2_0_2950-worker.noarch 1.4.1.2.3.2.0-2950.el6 HDP-2.3

And spark-shell is also showing 1.4.1. Weird!

So I am thinking Ambari is showing the wrong version somehow (not relating to the actual repos/packages on the cluster). We've had this problem before so maybe a bug? (the issue before was the version of Spark reported in HDP 2.2.4.2-2 in Stacks and Versions within the Admin menu was 1.2.0 and not 1.2.1 which was the actual version installed).

btw, the reason we need a specific version is we are about to release to market our product and it is in solution validation. They are expecting Spark 1.4.1 and that is what we have tested with so this was a bit of a surprise. Thankfully it seems like a minor issue in Ambari?

And going forward yes we definitely want keep up to date with Spark versions.

thanks,

Alan

[#4]

Yes as long as your application is leveraging correct library versions, you should be fine. In the long run, consider contacting support and resolve inconsistencies.

[#5]

I got confirmation from HW that HDP 2.3.2 is Spark 1.4.1 okay. The source of truth is the release notes.

So it looks like a bug in Ambari (maybe just with redhat7)

FYI here is the Ambari screenshot

[#6]

@Alan McShane Thanks for sharing this information. Did you get any information on Bug number?