Installing Spark2

[#1]

http://archive.cloudera.com/spark2/parcels/2.0.0.cloudera2/

How do I add this to my cluster conf files to install Spark2?

[#2]

@ujj

CM -> Administrator -> Settings -> Custom Service descriptor (CSD) Path
CM -> Hosts -> Parcel

Navigate to Custom Service Descriptor (CSD) & Parcel to understand the available options and refer the below link, it will help you to add the service

https://www.cloudera.com/documentation/enterprise/5-5-x/topics/cm_mc_addon_services.html

[#3]

I can't automate it at cluster bootstrap time?

[#4]

Went there, activated it, but I still don't see it as a choice when I Add Service. 

[#5]

Hi wkupersa,

Can you fully describe your scenario? Are you attempting to use Cloudera Director to create a new cluster including Spark 2, or do you have an existing cluster you'd like to add Spark 2 to? Or is it some other scenario? The additional information will help narrow down the possible issues.

Thanks,

Bill

[#6]

Thanks Bill,

I just built this cluster using CDH 5.11.0.  I installed Spark1.6.0 thought the wizard along with Yarn, ZooKeeper, and HDFS. I verified that Spark 1.6.0 worked and later added Hive as well.  I added the parcel configuration for Spark 2. I downloaded it, distributed it, and activated it. It appears as distributed and anctivated under parcels.  I then expected to see it in the list of services that I could add under the cluster's  "Add Service" option, but I don't.  I turned off "Validate Parcel Relations" to see if that would cause it to appear, but it didn't. 

[#7 · Accepted solution]

I was able to add it as a service after activating the parcel AND downloading the csd jar. I didn't realize that I needed both of these. I thought it was either or. 

[#8]

That's correct, cool that you solved it 🙂 A simple way to think about it is that a parcel contains the binary contents, and a CSD contains the metadata for a service.

[#9]

All operations are usefully.But do you restart the CM & Cloudera Manager Agent? And You will deploy the Client Configuration after above tasks. Even if you will have to upgrade the jre version.

[#10]

Hi,

I have follwed all the steps to install Spark2 in cloudera vm but getting following error when I am trying to Host spark2 parcel.

CDH (5.8 and higher) parcel required for SPARK2 (2.2.0.cloudera2-1.cdh5.12.0.p0.232957) is not available.

I am using CDH version : 5.12 and using http://archive.cloudera.com/spark2/parcels/2.2.0.cloudera2/  link for parcel.

Please help me with the configurtion.