Reply
Explorer
Posts: 17
Registered: ‎08-11-2014

failed to upgrade to spark2.3 on CM 5.10.1

hi

I've followed instruction from here - https://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html

 

Downloaded parcel, replaced csd jar, distributed, activated, restarted

 

but when restarting cloudera-scm-server I get in its log file /var/log/cloudera-scm-server/cloudera-scm-server.log:

2018-05-09 14:59:31,361 WARN main:com.cloudera.csd.components.CsdLocalRepository: Skipping [/opt/cloudera/csd/SPARK2_ON_YARN-2.3.0.cloudera2.jar]: { Could not read service descriptor for csd [SPARK2_ON_YARN-2.3.0.cloudera2]: Could not resolve type id 'provided' into a subtype of [simple type, class com.cloudera.csd.descriptors.parameters.Parameter<java.lang.Object>]
at [Source: [B@44b2dfa7; line: 124, column: 45] (through reference chain: ServiceDescriptor["parameters"]) }

 

and i cannot use the spark2 service in the cluster.

 

In file SPARK23/scripts/common.sh I see: local MIN_CM_VERSION=$(parse_version_string "5.14.0")

 

Should i really upgrade to latest CM version??

I prefer staying on current cm version until we decide to upgrade the whole lot.

 

Or is there another issue here?

 

Thanks in advance

Doron

Explorer
Posts: 17
Registered: ‎08-11-2014

Re: failed to upgrade to spark2.3 on CM 5.10.1

I've upgraded CM to latest (5.14.3) and it works.
although - i'd still like to know if it is supported on lower CM version or not.
Highlighted
Cloudera Employee
Posts: 58
Registered: ‎11-16-2015

Re: failed to upgrade to spark2.3 on CM 5.10.1

Hi @doronve

 

What you observed is correct. CDS 2.3 won't work with CM 5.10. The pre-requisites link is there in the same url you've followed:

 

https://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html

                                            ↓

  1. Check that all the software prerequisites are satisfied. If not, you might need to upgrade or install other software components first. See CDS Powered By Apache Spark Requirements for details

                                            ↓

 

 Cloudera Manager Versions

 

CDS Powered By Apache Spark Version Cloudera Manager Version

2.3 Release 2Cloudera Manager 5.11 and higher
2.3 Release 1Never officially released; if downloaded, do not use
2.2 Release 2Cloudera Manager 5.8.3, 5.9 and higher
2.2 Release 1Cloudera Manager 5.8.3, 5.9 and higher
2.1 Release 1Cloudera Manager 5.8.3, 5.9 and higher
2.0 Release 2Cloudera Manager 5.8.3, 5.9 and higher
2.0 Release 1Cloudera Manager 5.8.3, 5.9 and higher
Announcements