Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark 2.3 with Cloduera Version 6?

Solved Go to solution
Highlighted

Spark 2.3 with Cloduera Version 6?

New Contributor

Hi,

I'm currently CDH 5.14 with Spark2, which uses the Spark CDS 2.3 (release 4).
I plan on upgrading to Cloudera Version 6, which I know uses CDS 2.2.  It's my understanding that before the upgrade I will need to remove Spark 2 via Cloduera Manager 

"If you are using a CDS 2 minor version higher than the version of Spark 2 included in the CDH 6 release you are upgrading to (2.2 in CDH 6.0.0), you must remove your Spark 2 services from Cloudera Manager."
https://www.cloudera.com/documentation/enterprise/upgrade/topics/ug_spark_standalone.html#concept_al...

As part of the upgrade, Spark2 CDS 2.2 would be installed.  At this point am I able to then re download the Service Descriptor for 2.3 (release 4) and upgrade to that?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Spark 2.3 with Cloduera Version 6?

Expert Contributor
Spark2 now comes bundled with CDH thus for CDH 6.0 you are tied to the Spark 2.2 release. You will not be able to install Spark 2.3 separately. Please be aware you are facing a transitional issue from separate Spark2 distribution to CDH distribution. Future CDH versions will update Spark2 releases accordingly to the latest versions, thus this situation would not appear anymore. If Spark 2.3 is a requirement we recommend to wait for the C6.1 release instead.

View solution in original post

11 REPLIES 11
Highlighted

Re: Spark 2.3 with Cloduera Version 6?

Expert Contributor
Spark2 now comes bundled with CDH thus for CDH 6.0 you are tied to the Spark 2.2 release. You will not be able to install Spark 2.3 separately. Please be aware you are facing a transitional issue from separate Spark2 distribution to CDH distribution. Future CDH versions will update Spark2 releases accordingly to the latest versions, thus this situation would not appear anymore. If Spark 2.3 is a requirement we recommend to wait for the C6.1 release instead.

View solution in original post

Highlighted

Re: Spark 2.3 with Cloduera Version 6?

New Contributor

Thank you for the claftiaftion.  Saved me a world of pain.

it's not very clear from the "Upgrading Spark" page that you cannot complete the upgrade when using version 6.

https://www.cloudera.com/documentation/spark2/latest/topics/spark2_packaging.html

Thanks again.

Highlighted

Re: Spark 2.3 with Cloduera Version 6?

New Contributor

We also like to have Spark 2.3 with CDH6, is it scheduled for CDH 6.1?

 

Thx,

Michel

Highlighted

Re: Spark 2.3 with Cloduera Version 6?

Expert Contributor
Yes CDH 6.1 will have a Spark2 version upgrade.
Highlighted

Re: Spark 2.3 with Cloduera Version 6?

New Contributor

Ok thanks. Any delivery expectation for 6.1?

 

Regards,

Michel

Highlighted

Re: Spark 2.3 with Cloduera Version 6?

Expert Contributor

It will be release within the regular schedule. If you need exact dates then please contact your Cloudera sales representative for more details.

Highlighted

Re: Spark 2.3 with Cloduera Version 6?

Explorer

This same issue is still present on CDH 6.1. 

Re: Spark 2.3 with Cloduera Version 6?

Expert Contributor

Not exactly, this is not an issue at all but expected. It is what the error message says, the Spark 2.3 service is now (in C6) provided by the CDH parcel. There is no separate Spark2 parcel anymore, and you need to uninstall the Spark 2 parcel as advised in the doc chapter Migrating Apache Spark Before Upgrading to CDH 6

Highlighted

Re: Spark 2.3 with Cloduera Version 6?

Explorer

do you mean that  we can't replace  spark version of CDH6.x.x 

Don't have an account?
Coming from Hortonworks? Activate your account here