Support Questions

Find answers, ask questions, and share your expertise

Is it possible to use Spark 2.3 with CDH 6.0.x or 6.1.x?

Explorer

I am running CDH and CM 6.1.x and trying to install Spark 2.3- however after distributing the parcel, when I choose "activate" I get the following message:

4 REPLIES 4

Explorer

Actually, I see that the jar I am trying to use appears to be for cdh 5.13. Is there one for 6.x?

Explorer

I was able to get the parcel activated, but now Spark2 does not display under "Add Services", even after restarting via the instructions.

Cloudera Employee

 Spark 2.4 service is now (in C6.1) provided by the CDH parcel. There is no separate Spark2 parcel now. Please refer to the following link about Migrating Apache Spark Before Upgrading to CDH 6 :

https://docs.cloudera.com/documentation/enterprise/upgrade/topics/ug_spark_standalone.html#concept_p...

Cloudera Employee

Hi, 

 

CDH 6.1 package comes with spark 2.4 version. please see the below link for more information

 

https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_61_packaging.html

 

CDH 6 does not have separated parcels for Spark. it comes with CDH
 
Thanks
AK
Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.