I'm trying to get Spark 2 to work within the QuickStart vm for Docker. The version of the vm is recent, I pulled it down and installed it 2 days ago from the Cloudera website.
Following the directions, I have been able to download/deploy/activate the Spark 2 parcel using the Cloudera Manager Express successfully. But I think I have a problem with the gateway.
When I try to use the spark2-shell or pyspark2 from the command line I get the following message:
[root@quickstart /]# pyspark2 Failed to find client configuration. If this host is managed by Cloudera Manager, please install Spark Gateway Role on this host to run Spark Jobs. Otherwise, please configure the Spark dependencies correctly. [root@quickstart /]# spark2-shell Failed to find client configuration. If this host is managed by Cloudera Manager, please install Spark Gateway Role on this host to run Spark Jobs. Otherwise, please configure the Spark dependencies correctly. [root@quickstart /]#
Parcel configuration url: http://archive.cloudera.com/spark2/parcels/latest/
Cloudera parcel activated: CDH 5.13.0-1.cdh5.13.0.p0.29
Spark2 parcel activated: 2.4.0.cloudera2-1.cdh5.13.3.p0.1041012
When I look at CM-> Spark-2-> Instances It shows N/A for the gateway.
I'm hoping someone see's something I'm missing.
Thanks for the reply. I haven't had any success in getting the spark2 shell to run from the VM. I had to clean up the docker container because it was consuming a lot of disk space and we can't work without Spark 2.
I'm reinstalling the cloudera-quickstart-vm-5.13.0-0-beta-docker.tar.gz file and will update with my steps, incase I missed something.
What was the solution to this? I have the exact same problem in the Cloudera QuickStart VM for virtualbox.
I'm unable to add Spark 2 service in the cluster service list.
How did you run Migrate to Parcel script in docker?