Reply
New Contributor
Posts: 4
Registered: ‎02-26-2015

Is there an available guide on how to install Spark 1.4 (downloaded from spark.apache.org) ?

Hello,

 

We are planning to upgrade Spark from 1.3 to 1.4. Since this is not packaged with the latest CDH release, we are planning to manually donwload the Spark 1.4 from spark.apache.org but we are not sure on how we can install it on our existing Hadoop Cluster.

 

Kindly provide inputs if you have already tried this one.

 

Thank you!

 

Regards,

Mitch

Cloudera Employee
Posts: 366
Registered: ‎07-29-2013

Re: Is there an available guide on how to install Spark 1.4 (downloaded from spark.apache.org) ?

Probably 1.5 now?

You can simply use documentation from the main Spark project, like
http://spark.apache.org/docs/latest/hadoop-third-party-distributions.html
and http://spark.apache.org/docs/latest/running-on-yarn.html

It's been a while since I tried this, but I believe you want to grab
an assembly jar, ideally compiled for CDH, like a 1.5 snapshot build
from https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-assembly_2.10/1.5....

It's not guaranteed this works on CDH 5.4, but shouldn't be much if any problem.

I believe you run "SPARK_JAR=[the assembly .jar] spark-shell --master
yarn-client ..." for example to cause it to use this assembly for the
YARN app that you run.

Maybe someone else can provide more authoritative step by step
instructions but that's the essence of the idea.