Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Is there an available guide on how to install Spark 1.4 (downloaded from spark.apache.org) ?

Is there an available guide on how to install Spark 1.4 (downloaded from spark.apache.org) ?

New Contributor

Hello,

 

We are planning to upgrade Spark from 1.3 to 1.4. Since this is not packaged with the latest CDH release, we are planning to manually donwload the Spark 1.4 from spark.apache.org but we are not sure on how we can install it on our existing Hadoop Cluster.

 

Kindly provide inputs if you have already tried this one.

 

Thank you!

 

Regards,

Mitch

1 REPLY 1

Re: Is there an available guide on how to install Spark 1.4 (downloaded from spark.apache.org) ?

Master Collaborator
Probably 1.5 now?

You can simply use documentation from the main Spark project, like
http://spark.apache.org/docs/latest/hadoop-third-party-distributions.html
and http://spark.apache.org/docs/latest/running-on-yarn.html

It's been a while since I tried this, but I believe you want to grab
an assembly jar, ideally compiled for CDH, like a 1.5 snapshot build
from https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-assembly_2.10/1.5....

It's not guaranteed this works on CDH 5.4, but shouldn't be much if any problem.

I believe you run "SPARK_JAR=[the assembly .jar] spark-shell --master
yarn-client ..." for example to cause it to use this assembly for the
YARN app that you run.

Maybe someone else can provide more authoritative step by step
instructions but that's the essence of the idea.