question Re: Spark 2.3 with Cloduera Version 6? in Archives of Support Questions (Read Only)

Spark 2.3 with Cloduera Version 6?

chriswalton — Fri, 16 Sep 2022 13:52:58 GMT

Hi,

I'm currently CDH 5.14 with Spark2, which uses the Spark CDS 2.3 (release 4).
I plan on upgrading to Cloudera Version 6, which I know uses CDS 2.2. It's my understanding that before the upgrade I will need to remove Spark 2 via Cloduera Manager

"If you are using a CDS 2 minor version higher than the version of Spark 2 included in the CDH 6 release you are upgrading to (2.2 in CDH 6.0.0), you must remove your Spark 2 services from Cloudera Manager."
https://www.cloudera.com/documentation/enterprise/upgrade/topics/ug_spark_standalone.html#concept_alc_wpl_x2b

As part of the upgrade, Spark2 CDS 2.2 would be installed. At this point am I able to then re download the Service Descriptor for 2.3 (release 4) and upgrade to that?

Re: Spark 2.3 with Cloduera Version 6?

gzigldrum — Fri, 09 Nov 2018 13:33:22 GMT

Spark2 now comes bundled with CDH thus for CDH 6.0 you are tied to the Spark 2.2 release. You will not be able to install Spark 2.3 separately. Please be aware you are facing a transitional issue from separate Spark2 distribution to CDH distribution. Future CDH versions will update Spark2 releases accordingly to the latest versions, thus this situation would not appear anymore. If Spark 2.3 is a requirement we recommend to wait for the C6.1 release instead.

Re: Spark 2.3 with Cloduera Version 6?

chriswalton — Fri, 09 Nov 2018 13:42:49 GMT

Thank you for the claftiaftion. Saved me a world of pain.

it's not very clear from the "Upgrading Spark" page that you cannot complete the upgrade when using version 6.

https://www.cloudera.com/documentation/spark2/latest/topics/spark2_packaging.html

Thanks again.

Re: Spark 2.3 with Cloduera Version 6?

miezel — Fri, 09 Nov 2018 15:34:49 GMT

We also like to have Spark 2.3 with CDH6, is it scheduled for CDH 6.1?

Thx,

Michel

Re: Spark 2.3 with Cloduera Version 6?

gzigldrum — Mon, 12 Nov 2018 07:23:20 GMT

Yes CDH 6.1 will have a Spark2 version upgrade.

Re: Spark 2.3 with Cloduera Version 6?

miezel — Mon, 12 Nov 2018 07:36:25 GMT

Ok thanks. Any delivery expectation for 6.1?

Regards,

Michel

Re: Spark 2.3 with Cloduera Version 6?

gzigldrum — Mon, 12 Nov 2018 07:51:48 GMT

It will be release within the regular schedule. If you need exact dates then please contact your Cloudera sales representative for more details.

Re: Spark 2.3 with Cloduera Version 6?

krieger — Tue, 22 Oct 2019 21:55:52 GMT

This same issue is still present on CDH 6.1.

Re: Spark 2.3 with Cloduera Version 6?

gzigldrum — Wed, 23 Oct 2019 07:50:46 GMT

Not exactly, this is not an issue at all but expected. It is what the error message says, the Spark 2.3 service is now (in C6) provided by the CDH parcel. There is no separate Spark2 parcel anymore, and you need to uninstall the Spark 2 parcel as advised in the doc chapter Migrating Apache Spark Before Upgrading to CDH 6

Re: Spark 2.3 with Cloduera Version 6?

zdkd — Sat, 07 Mar 2020 10:25:55 GMT

can i replace the spark version from high to low? for example,cdh6.0 's build-in spark version is 2.2,can i replace it to 2.1

Re: Spark 2.3 with Cloduera Version 6?

zdkd — Sat, 07 Mar 2020 10:33:07 GMT

do you mean that we can't replace spark version of CDH6.x.x

Re: Spark 2.3 with Cloduera Version 6?

ask_bill_brooks — Sat, 07 Mar 2020 21:15:09 GMT

asked and answered here.

question Re: Spark 2.3 with Cloduera Version 6? in Archives of Support Questions (Read Only)

Spark 2.3 with Cloduera Version 6?

Re: Spark 2.3 with Cloduera Version 6?

Re: Spark 2.3 with Cloduera Version 6?

Re: Spark 2.3 with Cloduera Version 6?

Re: Spark 2.3 with Cloduera Version 6?

Re: Spark 2.3 with Cloduera Version 6?

Re: Spark 2.3 with Cloduera Version 6?

Re: Spark 2.3 with Cloduera Version 6?

Error

Re: Spark 2.3 with Cloduera Version 6?

Re: Spark 2.3 with Cloduera Version 6?

Re: Spark 2.3 with Cloduera Version 6?

Re: Spark 2.3 with Cloduera Version 6?