Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

failed to upgrade to spark2.3 on CM 5.10.1

failed to upgrade to spark2.3 on CM 5.10.1

Explorer

hi

I've followed instruction from here - https://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html

 

Downloaded parcel, replaced csd jar, distributed, activated, restarted

 

but when restarting cloudera-scm-server I get in its log file /var/log/cloudera-scm-server/cloudera-scm-server.log:

2018-05-09 14:59:31,361 WARN main:com.cloudera.csd.components.CsdLocalRepository: Skipping [/opt/cloudera/csd/SPARK2_ON_YARN-2.3.0.cloudera2.jar]: { Could not read service descriptor for csd [SPARK2_ON_YARN-2.3.0.cloudera2]: Could not resolve type id 'provided' into a subtype of [simple type, class com.cloudera.csd.descriptors.parameters.Parameter<java.lang.Object>]
at [Source: [B@44b2dfa7; line: 124, column: 45] (through reference chain: ServiceDescriptor["parameters"]) }

 

and i cannot use the spark2 service in the cluster.

 

In file SPARK23/scripts/common.sh I see: local MIN_CM_VERSION=$(parse_version_string "5.14.0")

 

Should i really upgrade to latest CM version??

I prefer staying on current cm version until we decide to upgrade the whole lot.

 

Or is there another issue here?

 

Thanks in advance

Doron

2 REPLIES 2

Re: failed to upgrade to spark2.3 on CM 5.10.1

Explorer
I've upgraded CM to latest (5.14.3) and it works.
although - i'd still like to know if it is supported on lower CM version or not.
Highlighted

Re: failed to upgrade to spark2.3 on CM 5.10.1

Expert Contributor

Hi @doronve

 

What you observed is correct. CDS 2.3 won't work with CM 5.10. The pre-requisites link is there in the same url you've followed:

 

https://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html

                                            ↓

  1. Check that all the software prerequisites are satisfied. If not, you might need to upgrade or install other software components first. See CDS Powered By Apache Spark Requirements for details

                                            ↓

 

 Cloudera Manager Versions

 

CDS Powered By Apache Spark Version Cloudera Manager Version

2.3 Release 2Cloudera Manager 5.11 and higher
2.3 Release 1Never officially released; if downloaded, do not use
2.2 Release 2Cloudera Manager 5.8.3, 5.9 and higher
2.2 Release 1Cloudera Manager 5.8.3, 5.9 and higher
2.1 Release 1Cloudera Manager 5.8.3, 5.9 and higher
2.0 Release 2Cloudera Manager 5.8.3, 5.9 and higher
2.0 Release 1Cloudera Manager 5.8.3, 5.9 and higher