Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark version different to expected in HDP 2.3.2

Solved Go to solution

Spark version different to expected in HDP 2.3.2

New Contributor

We have installed a fresh HDP 2.3.2 cluster and was expecting Spark 1.4.1 (as per build metadata, release notes etc.) but got Spark 1.5.2

Is this expected?

The repo was changed to 2.3.2 and only the redhat7 one was selected during installation..see screenshot.

thanks for your help, Alan

2509-capture.png

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Spark version different to expected in HDP 2.3.2

Mentor
@Alan McShane

you can confirm by going to the host and checking in /etc/yum.repos.d/ in HDP.repo file. Please double check whether repo is pointing to 2.3.4 and not 2.3.2. Spark 1.5.2 ships with HDP 2.3.4, Spark 1.4.1 ships with HDP 2.3.2. We just shipped Spark 1.6 with HDP 2.4. Why would you not want Spark 1.5.2 as that has newer features and stabilization fixes on top of 1.4.1.

View solution in original post

5 REPLIES 5
Highlighted

Re: Spark version different to expected in HDP 2.3.2

Mentor
@Alan McShane

you can confirm by going to the host and checking in /etc/yum.repos.d/ in HDP.repo file. Please double check whether repo is pointing to 2.3.4 and not 2.3.2. Spark 1.5.2 ships with HDP 2.3.4, Spark 1.4.1 ships with HDP 2.3.2. We just shipped Spark 1.6 with HDP 2.4. Why would you not want Spark 1.5.2 as that has newer features and stabilization fixes on top of 1.4.1.

View solution in original post

Highlighted

Re: Spark version different to expected in HDP 2.3.2

New Contributor

@Artem Ervits

Thanks for the quick reply :) The HDP.repo is saying it is definitely 2.3.2 okay.

Yum is also showing the packages are Spark 1.4.1 as expected.

spark.noarch  1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark-master.noarch  1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark-python.noarch  1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark-worker.noarch  1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark_2_3_2_0_2950.noarch  1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark_2_3_2_0_2950-master.noarch 1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark_2_3_2_0_2950-python.noarch 1.4.1.2.3.2.0-2950.el6 HDP-2.3
spark_2_3_2_0_2950-worker.noarch 1.4.1.2.3.2.0-2950.el6 HDP-2.3

And spark-shell is also showing 1.4.1. Weird!

So I am thinking Ambari is showing the wrong version somehow (not relating to the actual repos/packages on the cluster). We've had this problem before so maybe a bug? (the issue before was the version of Spark reported in HDP 2.2.4.2-2 in Stacks and Versions within the Admin menu was 1.2.0 and not 1.2.1 which was the actual version installed).

btw, the reason we need a specific version is we are about to release to market our product and it is in solution validation. They are expecting Spark 1.4.1 and that is what we have tested with so this was a bit of a surprise. Thankfully it seems like a minor issue in Ambari?

And going forward yes we definitely want keep up to date with Spark versions.

thanks,

Alan

Highlighted

Re: Spark version different to expected in HDP 2.3.2

Mentor

Yes as long as your application is leveraging correct library versions, you should be fine. In the long run, consider contacting support and resolve inconsistencies.

Highlighted

Re: Spark version different to expected in HDP 2.3.2

New Contributor

I got confirmation from HW that HDP 2.3.2 is Spark 1.4.1 okay. The source of truth is the release notes.

So it looks like a bug in Ambari (maybe just with redhat7)

FYI here is the Ambari screenshot

2607-hdp232-spark152.png

Highlighted

Re: Spark version different to expected in HDP 2.3.2

@Alan McShane Thanks for sharing this information. Did you get any information on Bug number?

Don't have an account?
Coming from Hortonworks? Activate your account here