question Installing Spark2 in Support Questions

Installing Spark2

ujj — Fri, 16 Sep 2022 11:22:08 GMT

http://archive.cloudera.com/spark2/parcels/2.0.0.cloudera2/

How do I add this to my cluster conf files to install Spark2?

Re: Installing Spark2

saranvisa — Thu, 30 Mar 2017 03:51:33 GMT

CM -> Administrator -> Settings -> Custom Service descriptor (CSD) Path
CM -> Hosts -> Parcel

Navigate to Custom Service Descriptor (CSD) & Parcel to understand the available options and refer the below link, it will help you to add the service

https://www.cloudera.com/documentation/enterprise/5-5-x/topics/cm_mc_addon_services.html

Re: Installing Spark2

ujj — Thu, 30 Mar 2017 04:19:57 GMT

I can't automate it at cluster bootstrap time?

Re: Installing Spark2

wkupersa — Mon, 15 May 2017 21:06:56 GMT

Went there, activated it, but I still don't see it as a choice when I Add Service.

Re: Installing Spark2

Bill Havanki — Mon, 15 May 2017 21:11:03 GMT

Hi wkupersa,

Can you fully describe your scenario? Are you attempting to use Cloudera Director to create a new cluster including Spark 2, or do you have an existing cluster you'd like to add Spark 2 to? Or is it some other scenario? The additional information will help narrow down the possible issues.

Thanks,

Bill

Re: Installing Spark2

wkupersa — Mon, 15 May 2017 21:32:12 GMT

Thanks Bill,

I just built this cluster using CDH 5.11.0. I installed Spark1.6.0 thought the wizard along with Yarn, ZooKeeper, and HDFS. I verified that Spark 1.6.0 worked and later added Hive as well. I added the parcel configuration for Spark 2. I downloaded it, distributed it, and activated it. It appears as distributed and anctivated under parcels. I then expected to see it in the list of services that I could add under the cluster's "Add Service" option, but I don't. I turned off "Validate Parcel Relations" to see if that would cause it to appear, but it didn't.

Re: Installing Spark2

wkupersa — Wed, 17 May 2017 22:05:53 GMT

I was able to add it as a service after activating the parcel AND downloading the csd jar. I didn't realize that I needed both of these. I thought it was either or.

Re: Installing Spark2

pmilani — Thu, 18 May 2017 13:47:33 GMT

That's correct, cool that you solved it 🙂 A simple way to think about it is that a parcel contains the binary contents, and a CSD contains the metadata for a service.

Re: Installing Spark2

pigtail — Tue, 20 Jun 2017 09:47:28 GMT

All operations are usefully.But do you restart the CM & Cloudera Manager Agent? And You will deploy the Client Configuration after above tasks. Even if you will have to upgrade the jre version.

Re: Installing Spark2

subhrajit — Thu, 15 Feb 2018 16:59:26 GMT

Hi,

I have follwed all the steps to install Spark2 in cloudera vm but getting following error when I am trying to Host spark2 parcel.

CDH (5.8 and higher) parcel required for SPARK2 (2.2.0.cloudera2-1.cdh5.12.0.p0.232957) is not available.

I am using CDH version : 5.12 and using http://archive.cloudera.com/spark2/parcels/2.2.0.cloudera2/ link for parcel.

Please help me with the configurtion.