Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark 2 in Cloudera QuickStart Docker VM not working

Highlighted

Spark 2 in Cloudera QuickStart Docker VM not working

New Contributor

Hello,

 

I'm trying to get Spark 2 to work within the QuickStart vm for Docker.  The version of the vm is recent, I pulled it down and installed it 2 days ago from the Cloudera website.

 

Following the directions, I have been able to download/deploy/activate the Spark 2 parcel using the Cloudera Manager Express successfully.  But I think I have a problem with the gateway.

 

When I try to use the spark2-shell or pyspark2 from the command line I get the following message:

 

[root@quickstart /]# pyspark2
Failed to find client configuration. If this host is managed by Cloudera Manager, please install Spark Gateway Role on this host to run Spark Jobs. Otherwise, please configure the Spark dependencies correctly.
[root@quickstart /]# spark2-shell 
Failed to find client configuration. If this host is managed by Cloudera Manager, please install Spark Gateway Role on this host to run Spark Jobs. Otherwise, please configure the Spark dependencies correctly.
[root@quickstart /]# 

  

Some details.

Parcel configuration url: http://archive.cloudera.com/spark2/parcels/latest/

Cloudera parcel activated: CDH 5.13.0-1.cdh5.13.0.p0.29

Spark2 parcel activated: 2.4.0.cloudera2-1.cdh5.13.3.p0.1041012

 

When I look at CM-> Spark-2-> Instances It shows N/A for the gateway.

 

Screenshot from 2019-05-15 12-32-26.png

Screenshot from 2019-05-15 12-34-24.png

 

I'm hoping someone see's something I'm missing. 

 

 

 

 

4 REPLIES 4

Re: Spark 2 in Cloudera QuickStart Docker VM not working

Rising Star
Hi Dean,

Gateway role is just for the client configuration, As it is not a service the state is denoted as N/A and it is expected behavior.

Do you facing this issue, Even after adding the Gateway node/ Deploying client configuration.

Thanks
Jerry

Re: Spark 2 in Cloudera QuickStart Docker VM not working

New Contributor

Hi Jerry,

 

Thanks for the reply.  I haven't had any success in getting the spark2 shell to run from the VM.  I had to clean up the docker container because it was consuming a lot of disk space and we can't work without Spark 2.

 

I'm reinstalling the cloudera-quickstart-vm-5.13.0-0-beta-docker.tar.gz file and will update with my steps, incase I missed something.

 

Re: Spark 2 in Cloudera QuickStart Docker VM not working

New Contributor

What was the solution to this? I have the exact same problem in the Cloudera QuickStart VM for virtualbox.

Re: Spark 2 in Cloudera QuickStart Docker VM not working

New Contributor

I'm unable to add Spark 2 service in the cluster service list. 
How did you run Migrate to Parcel script in docker?  

Don't have an account?
Coming from Hortonworks? Activate your account here