question Re: Spark 2 in Archives of Support Questions (Read Only)

Spark 2

dautovri — Fri, 16 Sep 2022 10:52:40 GMT

CDH (lower than 5.12) parcel required for SPARK2 (2.0.0.cloudera1-1.cdh5.7.0.p0.113931) is not available. How to solve this problem? I already copy SPARK2_ON_YARN-2.0.0.cloudera1.jar to the system and follow all steps from manuals.

Re: Spark 2

kchaitanya — Mon, 02 Jan 2017 09:00:35 GMT

Did you find any solution

Re: Spark 2

srowen — Mon, 02 Jan 2017 09:06:09 GMT

There is no CDH 5.12. Spark 2 is availabled as a CSD. Please follow the documented steps for installing it, which never includes manually copying JARs. http://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html

Re: Spark 2

kchaitanya — Mon, 02 Jan 2017 09:35:15 GMT

Followed same institutions. Downloaded CSD jar file and moved to /opt/cloudera/csd and restarted the server and management services. But no luck

Re: Spark 2

srowen — Mon, 02 Jan 2017 13:27:31 GMT

It sounds like you only finished step 2. You need to finish all of them.

Re: Spark 2

dautovri — Tue, 03 Jan 2017 01:53:18 GMT

"4.Download the Spark2 parcel, distribute the parcel to the hosts in your cluster, and activate the parcel. See Managing Parcels." I have the trouble with activation.

Re: Spark 2

kchaitanya — Tue, 03 Jan 2017 03:47:19 GMT

@sroweni did all, downloaded parcel and distributed to all nodes, while activating the parcel i'm getting CDH (lower than 5.12) parcel required for SPARK2 (2.0.0.cloudera1-1.cdh5.7.0.p0.113931) is not available

Re: Spark 2

dautovri — Tue, 03 Jan 2017 09:16:42 GMT

by Victor

figured it out, under parcel configurations, if I uncheck
Validate Parcel Relations

Re: Spark 2

kchaitanya — Tue, 03 Jan 2017 09:21:34 GMT

I hope it's not a good idea, as it says "Enforce that parcel dependencies are satisfied and conflicts are prevented when activating parcels. Parcel relations (Depends, Conflicts, and Replaces) can be defined the manifests of parcel repositories. Cloudera Manager can also enforce some default relations if none are defined in the manifest"

Anyway thanks for the tip 🙂

Re: Spark 2

hubbarja — Tue, 03 Jan 2017 16:15:08 GMT

Yes, this would typically not be recommened, but is a work around for a bug. This is fixed in CM 5.9, so when using 5.9 and newer, you should not need to disable parcel relation validation.

Re: Spark 2

MilesYao — Wed, 04 Jan 2017 22:33:24 GMT

We were able to install the official parcel. The only problem encountered was that all the signature files in the repository have extension .sha1. Our CM (5.8.3) were expecting .sha . Manually renaming it allowed the install to complete.

Re: Spark 2

Fawze — Wed, 08 Feb 2017 04:22:23 GMT

How i can know if the parcels files should has sha or sha1 signature?

When Spark2 will be part of CDH4 parcels, which version?

currently i have 1.6, when i add 2.0 will the spark history server be 2.0 also? if i used 2.0 parcels, which one will be the active one on the cluster 1.6 or 2.0?