question Re: Spark version different to expected in HDP 2.3.2 in Archives of Support Questions (Read Only)

Spark version different to expected in HDP 2.3.2

AlMcS — Fri, 16 Sep 2022 10:06:40 GMT

We have installed a fresh HDP 2.3.2 cluster and was expecting Spark 1.4.1 (as per build metadata, release notes etc.) but got Spark 1.5.2

Is this expected?

The repo was changed to 2.3.2 and only the redhat7 one was selected during installation..see screenshot.

thanks for your help, Alan

Re: Spark version different to expected in HDP 2.3.2

aervits — Wed, 02 Mar 2016 08:31:55 GMT

@Alan McShane

you can confirm by going to the host and checking in /etc/yum.repos.d/ in HDP.repo file. Please double check whether repo is pointing to 2.3.4 and not 2.3.2. Spark 1.5.2 ships with HDP 2.3.4, Spark 1.4.1 ships with HDP 2.3.2. We just shipped Spark 1.6 with HDP 2.4. Why would you not want Spark 1.5.2 as that has newer features and stabilization fixes on top of 1.4.1.

Re: Spark version different to expected in HDP 2.3.2

AlMcS — Wed, 02 Mar 2016 20:27:07 GMT

@Artem Ervits

Thanks for the quick reply 🙂 The HDP.repo is saying it is definitely 2.3.2 okay.

Yum is also showing the packages are Spark 1.4.1 as expected.

spark.noarch  1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark-master.noarch  1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark-python.noarch  1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark-worker.noarch  1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark_2_3_2_0_2950.noarch  1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark_2_3_2_0_2950-master.noarch 1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark_2_3_2_0_2950-python.noarch 1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark_2_3_2_0_2950-worker.noarch 1.4.1.2.3.2.0-2950.el6 HDP-2.3

And spark-shell is also showing 1.4.1. Weird!

So I am thinking Ambari is showing the wrong version somehow (not relating to the actual repos/packages on the cluster). We've had this problem before so maybe a bug? (the issue before was the version of Spark reported in HDP 2.2.4.2-2 in Stacks and Versions within the Admin menu was 1.2.0 and not 1.2.1 which was the actual version installed).

btw, the reason we need a specific version is we are about to release to market our product and it is in solution validation. They are expecting Spark 1.4.1 and that is what we have tested with so this was a bit of a surprise. Thankfully it seems like a minor issue in Ambari?

And going forward yes we definitely want keep up to date with Spark versions.

thanks,

Alan

Re: Spark version different to expected in HDP 2.3.2

aervits — Thu, 03 Mar 2016 02:03:06 GMT

Yes as long as your application is leveraging correct library versions, you should be fine. In the long run, consider contacting support and resolve inconsistencies.

Re: Spark version different to expected in HDP 2.3.2

AlMcS — Mon, 19 Aug 2019 11:58:59 GMT

I got confirmation from HW that HDP 2.3.2 is Spark 1.4.1 okay. The source of truth is the release notes.

So it looks like a bug in Ambari (maybe just with redhat7)

FYI here is the Ambari screenshot

Re: Spark version different to expected in HDP 2.3.2

nsabharwal — Sat, 05 Mar 2016 11:10:57 GMT

@Alan McShane Thanks for sharing this information. Did you get any information on Bug number?