Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Spark version different to expected in HDP 2.3.2

avatar
New Contributor

We have installed a fresh HDP 2.3.2 cluster and was expecting Spark 1.4.1 (as per build metadata, release notes etc.) but got Spark 1.5.2

Is this expected?

The repo was changed to 2.3.2 and only the redhat7 one was selected during installation..see screenshot.

thanks for your help, Alan

2509-capture.png

1 ACCEPTED SOLUTION

avatar
Master Mentor
@Alan McShane

you can confirm by going to the host and checking in /etc/yum.repos.d/ in HDP.repo file. Please double check whether repo is pointing to 2.3.4 and not 2.3.2. Spark 1.5.2 ships with HDP 2.3.4, Spark 1.4.1 ships with HDP 2.3.2. We just shipped Spark 1.6 with HDP 2.4. Why would you not want Spark 1.5.2 as that has newer features and stabilization fixes on top of 1.4.1.

View solution in original post

5 REPLIES 5

avatar
Master Mentor
@Alan McShane

you can confirm by going to the host and checking in /etc/yum.repos.d/ in HDP.repo file. Please double check whether repo is pointing to 2.3.4 and not 2.3.2. Spark 1.5.2 ships with HDP 2.3.4, Spark 1.4.1 ships with HDP 2.3.2. We just shipped Spark 1.6 with HDP 2.4. Why would you not want Spark 1.5.2 as that has newer features and stabilization fixes on top of 1.4.1.

avatar
New Contributor

@Artem Ervits

Thanks for the quick reply 🙂 The HDP.repo is saying it is definitely 2.3.2 okay.

Yum is also showing the packages are Spark 1.4.1 as expected.

spark.noarch  1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark-master.noarch  1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark-python.noarch  1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark-worker.noarch  1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark_2_3_2_0_2950.noarch  1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark_2_3_2_0_2950-master.noarch 1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark_2_3_2_0_2950-python.noarch 1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark_2_3_2_0_2950-worker.noarch 1.4.1.2.3.2.0-2950.el6 HDP-2.3

And spark-shell is also showing 1.4.1. Weird!

So I am thinking Ambari is showing the wrong version somehow (not relating to the actual repos/packages on the cluster). We've had this problem before so maybe a bug? (the issue before was the version of Spark reported in HDP 2.2.4.2-2 in Stacks and Versions within the Admin menu was 1.2.0 and not 1.2.1 which was the actual version installed).

btw, the reason we need a specific version is we are about to release to market our product and it is in solution validation. They are expecting Spark 1.4.1 and that is what we have tested with so this was a bit of a surprise. Thankfully it seems like a minor issue in Ambari?

And going forward yes we definitely want keep up to date with Spark versions.

thanks,

Alan

avatar
Master Mentor

Yes as long as your application is leveraging correct library versions, you should be fine. In the long run, consider contacting support and resolve inconsistencies.

avatar
New Contributor

I got confirmation from HW that HDP 2.3.2 is Spark 1.4.1 okay. The source of truth is the release notes.

So it looks like a bug in Ambari (maybe just with redhat7)

FYI here is the Ambari screenshot

2607-hdp232-spark152.png

avatar
Master Mentor

@Alan McShane Thanks for sharing this information. Did you get any information on Bug number?