Reply
Highlighted
New Contributor
Posts: 2
Registered: ‎05-15-2019

Spark 2 in Cloudera QuickStart Docker VM not working

Hello,

 

I'm trying to get Spark 2 to work within the QuickStart vm for Docker.  The version of the vm is recent, I pulled it down and installed it 2 days ago from the Cloudera website.

 

Following the directions, I have been able to download/deploy/activate the Spark 2 parcel using the Cloudera Manager Express successfully.  But I think I have a problem with the gateway.

 

When I try to use the spark2-shell or pyspark2 from the command line I get the following message:

 

[root@quickstart /]# pyspark2
Failed to find client configuration. If this host is managed by Cloudera Manager, please install Spark Gateway Role on this host to run Spark Jobs. Otherwise, please configure the Spark dependencies correctly.
[root@quickstart /]# spark2-shell 
Failed to find client configuration. If this host is managed by Cloudera Manager, please install Spark Gateway Role on this host to run Spark Jobs. Otherwise, please configure the Spark dependencies correctly.
[root@quickstart /]# 

  

Some details.

Parcel configuration url: http://archive.cloudera.com/spark2/parcels/latest/

Cloudera parcel activated: CDH 5.13.0-1.cdh5.13.0.p0.29

Spark2 parcel activated: 2.4.0.cloudera2-1.cdh5.13.3.p0.1041012

 

When I look at CM-> Spark-2-> Instances It shows N/A for the gateway.

 

Screenshot from 2019-05-15 12-32-26.png

Screenshot from 2019-05-15 12-34-24.png

 

I'm hoping someone see's something I'm missing. 

 

 

 

 

Posts: 49
Topics: 0
Kudos: 13
Solutions: 5
Registered: ‎07-30-2018

Re: Spark 2 in Cloudera QuickStart Docker VM not working

Hi Dean,

Gateway role is just for the client configuration, As it is not a service the state is denoted as N/A and it is expected behavior.

Do you facing this issue, Even after adding the Gateway node/ Deploying client configuration.

Thanks
Jerry
New Contributor
Posts: 2
Registered: ‎05-15-2019

Re: Spark 2 in Cloudera QuickStart Docker VM not working

[ Edited ]

Hi Jerry,

 

Thanks for the reply.  I haven't had any success in getting the spark2 shell to run from the VM.  I had to clean up the docker container because it was consuming a lot of disk space and we can't work without Spark 2.

 

I'm reinstalling the cloudera-quickstart-vm-5.13.0-0-beta-docker.tar.gz file and will update with my steps, incase I missed something.