question Re: Cloudera VM Free to use Apache Hadoop with Spark in Archives of Support Questions (Read Only)

Cloudera VM Free to use Apache Hadoop with Spark

prodgers125 — Fri, 16 Sep 2022 10:18:32 GMT

Hi,

There exists some free Vitual Machine to use Apache Hadoop and Spark? I need to do some taks with HDFS and Hive and next some analysis with Spark.

Thanks!

Re: Cloudera VM Free to use Apache Hadoop with Spark

Sean — Wed, 11 May 2016 19:24:39 GMT

Yep, right here: http://www.cloudera.com/downloads/quickstart_vms/5-7.html

Re: Cloudera VM Free to use Apache Hadoop with Spark

prodgers125 — Wed, 11 May 2016 19:26:15 GMT

Sean, just many thanks for your response. This machine have pySpark ???

Re: Cloudera VM Free to use Apache Hadoop with Spark

Sean — Wed, 11 May 2016 19:29:31 GMT

Yes it does.

Re: Cloudera VM Free to use Apache Hadoop with Spark

Tmang — Wed, 18 May 2016 13:16:41 GMT

Actually, I have a problem with Spark. I don't have Spark listed among installed applications. I can see a Spark folder, however clicking it I get a message, That server is too busy, and it can't connect to ....:18080

I wonder it's just my experience. Everything else works fine, I can perform training tasks. What can cause it? I thought Spark is not a part of Cloudera VM.

Re: Cloudera VM Free to use Apache Hadoop with Spark

Tmang — Wed, 18 May 2016 13:43:25 GMT

Ok, I missed, I have 5.5. I have to download 5.7

Re: Cloudera VM Free to use Apache Hadoop with Spark

Tmang — Sat, 21 May 2016 23:23:50 GMT

I downloaded Clouera Quickstart VM 5-7, it doesn't have Spark. Do you have any recommendation where to find istructions how to install spark on hadoop?

[cloudera@quickstart ~]$ hadoop fs -ls /user/
Found 9 items
drwxr-xr-x - cloudera cloudera 0 2016-05-21 16:05 /user/cloudera
drwxr-xr-x - hdfs supergroup 0 2016-05-21 16:05 /user/hdfs
drwxr-xr-x - mapred hadoop 0 2016-04-06 01:25 /user/history
drwxrwxrwx - hive supergroup 0 2016-04-06 01:27 /user/hive
drwxrwxrwx - hue supergroup 0 2016-05-21 16:07 /user/hue
drwxrwxrwx - jenkins supergroup 0 2016-04-06 01:25 /user/jenkins
drwxrwxrwx - oozie supergroup 0 2016-04-06 01:26 /user/oozie
drwxrwxrwx - root supergroup 0 2016-04-06 01:25 /user/root
drwxr-xr-x - hdfs supergroup 0 2016-04-06 01:27 /user/spark
[cloudera@quickstart ~]$ hadoop fs -ls /user/spark/
Found 1 items
drwxr-xr-x - spark supergroup 0 2016-05-21 16:17 /user/spark/applicationHistory
[cloudera@quickstart ~]$ hadoop fs -ls /user/spark/applicationHistory/
[cloudera@quickstart ~]$

Re: Cloudera VM Free to use Apache Hadoop with Spark

Sean — Mon, 23 May 2016 14:44:19 GMT

Spark is installed in the 5.7 VM. We support Spark-on-YARN, and spark-shell
and pyspark are both on the PATH.

Re: Cloudera VM Free to use Apache Hadoop with Spark

Tmang — Mon, 23 May 2016 16:47:37 GMT

I downloaded on 05-20-2016. As you can see there's no spark listed istalled, only empty directory. Can anybody check/verify? Or where to find the instructions to instal on VM? Thank you.

Re: Cloudera VM Free to use Apache Hadoop with Spark

Sean — Mon, 23 May 2016 17:53:25 GMT

You're looking in HDFS directories - I expect those to be blank unless you've loaded some data to those directories or run some jobs. On the 5.7 VM I just successfully ran some Spark code by typing `pyspark` on the command-line or `spark-shell --master yarn-client` for the Scala shell. I confirmed that the spark-submit and spark-executor commands are also on the PATH.

Re: Cloudera VM Free to use Apache Hadoop with Spark

Sean — Mon, 23 May 2016 17:57:45 GMT

Here's a screenshot from my VM if this helps make it clearer what I'm talking about.

Re: Cloudera VM Free to use Apache Hadoop with Spark

Tmang — Mon, 23 May 2016 19:36:19 GMT

Ok, thank you. I'll try later. By the way can you open a spark folder from hue? I have a message that application is not installed, any other folders I can open. At least in 5-5 i had more directories for spark :-). That's why I didn't try any spark shells from a command line. I decided there was nothing there.

Re: Cloudera VM Free to use Apache Hadoop with Spark

Tmang — Mon, 23 May 2016 23:25:27 GMT

Thank you, I have spark! But what do I do wrong?

From localhost:4040/jobs I can see a job run, but from main menu I have no completed applications found.

Thank you.