Cloudera VM Free to use Apache Hadoop with Spark

[#1]

Hi,

There exists some free Vitual Machine to use Apache Hadoop and Spark? I need to do some taks with HDFS and Hive and next some analysis with Spark.

Thanks!

[#2 · Accepted solution]

Yep, right here: http://www.cloudera.com/downloads/quickstart_vms/5-7.html

[#3]

Sean, just many thanks for your response. This machine have pySpark ???

[#4]

Yes it does.

[#5]

Actually, I have a problem with Spark. I don't have Spark listed among installed applications. I can see a Spark folder, however clicking it  I get a message, That server is too busy, and it can't connect to ....:18080

I wonder it's just my experience. Everything else  works fine, I can perform training tasks. What can cause it? I thought Spark is not a part of Cloudera VM.

[#6]

Ok, I missed, I have 5.5. I have to download 5.7

[#7]

I downloaded Clouera Quickstart VM 5-7, it doesn't have Spark. Do you have any recommendation where to find istructions how to install spark on hadoop? 

[cloudera@quickstart ~]$ hadoop fs -ls /user/
Found 9 items
drwxr-xr-x - cloudera cloudera 0 2016-05-21 16:05 /user/cloudera
drwxr-xr-x - hdfs supergroup 0 2016-05-21 16:05 /user/hdfs
drwxr-xr-x - mapred hadoop 0 2016-04-06 01:25 /user/history
drwxrwxrwx - hive supergroup 0 2016-04-06 01:27 /user/hive
drwxrwxrwx - hue supergroup 0 2016-05-21 16:07 /user/hue
drwxrwxrwx - jenkins supergroup 0 2016-04-06 01:25 /user/jenkins
drwxrwxrwx - oozie supergroup 0 2016-04-06 01:26 /user/oozie
drwxrwxrwx - root supergroup 0 2016-04-06 01:25 /user/root
drwxr-xr-x - hdfs supergroup 0 2016-04-06 01:27 /user/spark
[cloudera@quickstart ~]$ hadoop fs -ls /user/spark/
Found 1 items
drwxr-xr-x - spark supergroup 0 2016-05-21 16:17 /user/spark/applicationHistory
[cloudera@quickstart ~]$ hadoop fs -ls /user/spark/applicationHistory/
[cloudera@quickstart ~]$

[#8]

Spark is installed in the 5.7 VM. We support Spark-on-YARN, and spark-shell
and pyspark are both on the PATH.

[#9]

I downloaded on 05-20-2016. As you can see there's no spark listed istalled, only empty directory. Can anybody check/verify? Or where to find the instructions to instal on VM? Thank you.

[#10]

You're looking in HDFS directories - I expect those to be blank unless you've loaded some data to those directories or run some jobs. On the 5.7 VM I just successfully ran some Spark code by typing `pyspark` on the command-line or `spark-shell --master yarn-client` for the Scala shell. I confirmed that the spark-submit and spark-executor commands are also on the PATH.

[#11]

Here's a screenshot from my VM if this helps make it clearer what I'm talking about.

[#12]

Ok, thank you. I'll try later. By the way can you open a spark folder from hue? I have a message that application is not installed, any other folders I can open. At least in 5-5 i had more directories for spark :-). That's why I didn't try any spark shells from a command line. I decided there was nothing there.

[#13]

Thank you, I have spark! But what do I do wrong?

From localhost:4040/jobs I can see a job run, but from main menu I have no completed applications found.

Thank you.