I'm currently CDH 5.14 with Spark2, which uses the Spark CDS 2.3 (release 4).
I plan on upgrading to Cloudera Version 6, which I know uses CDS 2.2. It's my understanding that before the upgrade I will need to remove Spark 2 via Cloduera Manager
"If you are using a CDS 2 minor version higher than the version of Spark 2 included in the CDH 6 release you are upgrading to (2.2 in CDH 6.0.0), you must remove your Spark 2 services from Cloudera Manager."
As part of the upgrade, Spark2 CDS 2.2 would be installed. At this point am I able to then re download the Service Descriptor for 2.3 (release 4) and upgrade to that?
Thank you for the claftiaftion. Saved me a world of pain.
it's not very clear from the "Upgrading Spark" page that you cannot complete the upgrade when using version 6.
It will be release within the regular schedule. If you need exact dates then please contact your Cloudera sales representative for more details.
This same issue is still present on CDH 6.1.
Not exactly, this is not an issue at all but expected. It is what the error message says, the Spark 2.3 service is now (in C6) provided by the CDH parcel. There is no separate Spark2 parcel anymore, and you need to uninstall the Spark 2 parcel as advised in the doc chapter Migrating Apache Spark Before Upgrading to CDH 6