Support Questions

Find answers, ask questions, and share your expertise

Spark2 service not found in cloudera manager

Explorer

I am using cloudera 5.10 and trying to install spark 2.3. I have csd and activated spark2 parcel. Also did a restart of cloudera server and  cloudera manager service. Still can't see the spark2 service. Before that I Deleted the service from Cloudera manager and trying to reinstall and found the  spark2 service is missing.

Could you please tell me what is wrong?

9 REPLIES 9

Cloudera Employee

I would suggest you to check Cloudera Manager's log.

If the CSD is correct, after restarting CM, there will be logs showing CM has recognized the CSD.

Then add the Spark2 parcel via CM UI.

Then you should be able to download the Spark2 parcel, distribute and activate it.

 

Then you should see Spark2 listed when you try to add new service via CM UI.

Explorer

HI,
Thanks for your reply.
I have checked  the logs and found this warning.
WARN main:com.cloudera.csd.components.CsdLocalRepository: Skipping [/opt/cloudera/csd/SPARK2_ON_YARN-2.3.0.cloudera4.jar]: { Could not read service descriptor for csd [SPARK2_ON_YARN-2.3.0.cloudera4]: Could not resolve type id 'provided' into a subtype of [simple type, class com.cloudera.csd.descriptors.parameters.Parameter<java.lang.Object>]

Explorer

Also The spark2 service was added earlier using parcel only as I haven't copied the csd file earlier.  I deleted the service rom cloudera manager and now I can't see the service. Do we need to perform some  more cleanup of service even if we delete from CM?

Cloudera Employee

I don't understand what you are trying to say.

If you have deployed the Spark 2 Gateway Role once, then the CSD file is correct.

If you have removed Spark2 via CM UI already, of course you can't find it.

 

Cloudera Employee
Well, then I believe it's not the correct CSD file...

Explorer

Thanks @ShounenG . Want to ask  how I was able to install spark2 from services  when I don't have csd only parcels. Now I deleted the spark2 service and installed the csd and activated parcels again. But now I can't see the spark2 service to install.
Is this that spark2 service deletion wasn't done properly?

Cloudera Employee

No, for me it doesn't sound like the deletion is not completely, but the CSD is preventing CM to recognize Spark 2.

New Contributor

Please can you make sure the CDS is in correct path and file perission are good. scm user account ( typically cloudera-scm) should be able to access it 

 

 

regards

Jay 

 

Explorer

@Jaydeep_Ta The permissions are in place and owner is cloudera-scm user. It have 644 permission. I tried to install the same  on 5.16 and it is working without any  issues. Only 5.10 is having issue.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.