Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Install cloudera spark 2

Install cloudera spark 2

New Contributor

Hey

 

Up until now we submited spark 1.6 jobs to our yarn cluster. Recently we added spark 2 to our nodes using cloudera.

 

We are using in house scheduling system which create docker machine for each job. These dockers spark-submit jobs to the cluster. 

To do it we install spark 1.6 cloudera release using yum install(on inhouse docker):

ADD cloudera-cdh$CDH_VERSION.repo      /etc/yum.repos.d/cloudera-cdh$CDH_VERSION.repo
yum install spark-core

This code installed spark 1.6 cloudera packages (spark-core-1.6.0+cdh5.12.1+530-1.cdh5.12.1.p0.3.el6.noarch ).

 

Now we want to do the same with spark2 which means that we want to install cloudera spark2 on our docker and submit jobs to cluster.

 

To my understanding there is no spark 2.3.0 pacakges only  

SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el5.parcel

There is also no installer, I know that the parcel is just a kind of tar but I don't have any idea how to install it correctly.

I there any installation process to the parcel(without cloudera agent)

1 REPLY 1

Re: Install cloudera spark 2

Master Guru
You may find success in using an unpacked parcel (tar xf file.parcel) if you pre-place the config files it relies on at the right locations (/etc/hadoop/conf, /etc/spark2/conf, etc. dirs and contents, obtainable from an existing CM-managed gateway host) and place the PARCEL_EXTRACT_ROOT/bin/ on the global path.

Once setup, try running the spark2-shell from SPARK2-PARCEL-EXTRACT-ROOT/bin/ level.

Note also that the Spark2 parcel relies on presence of CDH parcel, so both will need to be present.