Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: The Cloudera Community will undergo maintenance on Saturday, August 17 at 12:00am PDT. See more info here.

Spark 2 in Cloudera QuickStart Docker VM not working

Spark 2 in Cloudera QuickStart Docker VM not working

New Contributor

Hello,

 

I'm trying to get Spark 2 to work within the QuickStart vm for Docker.  The version of the vm is recent, I pulled it down and installed it 2 days ago from the Cloudera website.

 

Following the directions, I have been able to download/deploy/activate the Spark 2 parcel using the Cloudera Manager Express successfully.  But I think I have a problem with the gateway.

 

When I try to use the spark2-shell or pyspark2 from the command line I get the following message:

 

[root@quickstart /]# pyspark2
Failed to find client configuration. If this host is managed by Cloudera Manager, please install Spark Gateway Role on this host to run Spark Jobs. Otherwise, please configure the Spark dependencies correctly.
[root@quickstart /]# spark2-shell 
Failed to find client configuration. If this host is managed by Cloudera Manager, please install Spark Gateway Role on this host to run Spark Jobs. Otherwise, please configure the Spark dependencies correctly.
[root@quickstart /]# 

  

Some details.

Parcel configuration url: http://archive.cloudera.com/spark2/parcels/latest/

Cloudera parcel activated: CDH 5.13.0-1.cdh5.13.0.p0.29

Spark2 parcel activated: 2.4.0.cloudera2-1.cdh5.13.3.p0.1041012

 

When I look at CM-> Spark-2-> Instances It shows N/A for the gateway.

 

Screenshot from 2019-05-15 12-32-26.png

Screenshot from 2019-05-15 12-34-24.png

 

I'm hoping someone see's something I'm missing. 

 

 

 

 

2 REPLIES 2

Re: Spark 2 in Cloudera QuickStart Docker VM not working

Rising Star
Hi Dean,

Gateway role is just for the client configuration, As it is not a service the state is denoted as N/A and it is expected behavior.

Do you facing this issue, Even after adding the Gateway node/ Deploying client configuration.

Thanks
Jerry
Highlighted

Re: Spark 2 in Cloudera QuickStart Docker VM not working

New Contributor

Hi Jerry,

 

Thanks for the reply.  I haven't had any success in getting the spark2 shell to run from the VM.  I had to clean up the docker container because it was consuming a lot of disk space and we can't work without Spark 2.

 

I'm reinstalling the cloudera-quickstart-vm-5.13.0-0-beta-docker.tar.gz file and will update with my steps, incase I missed something.