Posts: 7
Registered: ‎07-31-2017

Spark2 on CDH 5.11

We have a CDH 5.11 setup without cloudera manager.

We wanted to install Spark 2.1 on it, and found CDH distribution of Spark.

However the nearest version available is SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904-el7.parcel . After this the next version is SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354-el7.parcel which is higher than 5.11


After extracting the parcel file and placing the folders at correct location, our jobs arent running smoothly as they were running on spark 1.6


The spark 2 steps are of my own as i didnt get any document on installing CDH spark manually.


1) Is CDH Spark 2.1 supported on CDH 5.11 ?

2) Can I get proper steps on this ?

Posts: 642
Topics: 3
Kudos: 103
Solutions: 67
Registered: ‎08-16-2016

Re: Spark2 on CDH 5.11

The versioning for Spark2 is odd and I won't try to explain as I don't understand it. Spark2 is separate from CDH though. I have Spark2 2.1 installed on CDH 5.11.1 and will be upgrading to Spark2 2.2 soon. I haven't seen any dependency information between Spark2 and CDH.

Anyway, your issue is that Spark2 can only be installed using parcels which are specific to CM. So without CM you are not going to be able to install Spark2 from Cloudera. In theory you could get the packages from Apache or elsewhere and manually install it.