Support Questions

Find answers, ask questions, and share your expertise

failed to upgrade to spark2.3 on CM 5.10.1



I've followed instruction from here -


Downloaded parcel, replaced csd jar, distributed, activated, restarted


but when restarting cloudera-scm-server I get in its log file /var/log/cloudera-scm-server/cloudera-scm-server.log:

2018-05-09 14:59:31,361 WARN main:com.cloudera.csd.components.CsdLocalRepository: Skipping [/opt/cloudera/csd/SPARK2_ON_YARN-2.3.0.cloudera2.jar]: { Could not read service descriptor for csd [SPARK2_ON_YARN-2.3.0.cloudera2]: Could not resolve type id 'provided' into a subtype of [simple type, class com.cloudera.csd.descriptors.parameters.Parameter<java.lang.Object>]
at [Source: [B@44b2dfa7; line: 124, column: 45] (through reference chain: ServiceDescriptor["parameters"]) }


and i cannot use the spark2 service in the cluster.


In file SPARK23/scripts/ I see: local MIN_CM_VERSION=$(parse_version_string "5.14.0")


Should i really upgrade to latest CM version??

I prefer staying on current cm version until we decide to upgrade the whole lot.


Or is there another issue here?


Thanks in advance



I've upgraded CM to latest (5.14.3) and it works.
although - i'd still like to know if it is supported on lower CM version or not.

Expert Contributor

Hi @doronve


What you observed is correct. CDS 2.3 won't work with CM 5.10. The pre-requisites link is there in the same url you've followed:


  1. Check that all the software prerequisites are satisfied. If not, you might need to upgrade or install other software components first. See CDS Powered By Apache Spark Requirements for details



 Cloudera Manager Versions


CDS Powered By Apache Spark Version Cloudera Manager Version

2.3 Release 2Cloudera Manager 5.11 and higher
2.3 Release 1Never officially released; if downloaded, do not use
2.2 Release 2Cloudera Manager 5.8.3, 5.9 and higher
2.2 Release 1Cloudera Manager 5.8.3, 5.9 and higher
2.1 Release 1Cloudera Manager 5.8.3, 5.9 and higher
2.0 Release 2Cloudera Manager 5.8.3, 5.9 and higher
2.0 Release 1Cloudera Manager 5.8.3, 5.9 and higher
Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.