Reply
New Contributor
Posts: 1
Registered: ‎02-06-2019

Spark 2.4 on CDH

Hi,

 

I'm currently running a cluster with Cloudera Enterprise 5.16.1 and I've upgraded it to use Spark 2.2. All is working as expected.

 

I now realise there's a feature in Spark 2.4 I'm keen to use but can't see an easy way to update my cluster since the latest CDS version available is 2.3 (https://www.cloudera.com/documentation/spark2/latest/topics/spark2_packaging.html#versions)

 

Is there an alternative way to get Spark 2.4 installed on the cluster or am I missing something obvious?

 

Thanks

Cloudera Employee
Posts: 739
Registered: ‎03-23-2015

Re: Spark 2.4 on CDH

No, you can't upgrade single component individually outside of CDH, it is unsupported.

Which specific feature that you are after? It might be included in CDH, as the version in CDH and upstream are different, we backport lots of fixes into our version, even though it is behind the version number compared with upstream.