Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark2 on CDH 5.11

Spark2 on CDH 5.11

Explorer

We have a CDH 5.11 setup without cloudera manager.

We wanted to install Spark 2.1 on it, and found CDH distribution of Spark.

However the nearest version available is SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904-el7.parcel . After this the next version is SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354-el7.parcel which is higher than 5.11

 

After extracting the parcel file and placing the folders at correct location, our jobs arent running smoothly as they were running on spark 1.6

 

The spark 2 steps are of my own as i didnt get any document on installing CDH spark manually.

 

1) Is CDH Spark 2.1 supported on CDH 5.11 ?

2) Can I get proper steps on this ?

1 REPLY 1
Highlighted

Re: Spark2 on CDH 5.11

Champion
The versioning for Spark2 is odd and I won't try to explain as I don't understand it. Spark2 is separate from CDH though. I have Spark2 2.1 installed on CDH 5.11.1 and will be upgrading to Spark2 2.2 soon. I haven't seen any dependency information between Spark2 and CDH.

Anyway, your issue is that Spark2 can only be installed using parcels which are specific to CM. So without CM you are not going to be able to install Spark2 from Cloudera. In theory you could get the packages from Apache or elsewhere and manually install it.