04-01-2019 07:04 AM - last edited on 04-01-2019 08:41 AM by cjervis
Up until now we submited spark 1.6 jobs to our yarn cluster. Recently we added spark 2 to our nodes using cloudera.
We are using in house scheduling system which create docker machine for each job. These dockers spark-submit jobs to the cluster.
To do it we install spark 1.6 cloudera release using yum install(on inhouse docker):
ADD cloudera-cdh$CDH_VERSION.repo /etc/yum.repos.d/cloudera-cdh$CDH_VERSION.repo
yum install spark-core
This code installed spark 1.6 cloudera packages (spark-core-1.6.0+cdh5.12.1+530-1.cdh5.12.1.p0.3.el6.noarch ).
Now we want to do the same with spark2 which means that we want to install cloudera spark2 on our docker and submit jobs to cluster.
To my understanding there is no spark 2.3.0 pacakges only
There is also no installer, I know that the parcel is just a kind of tar but I don't have any idea how to install it correctly.
I there any installation process to the parcel(without cloudera agent)
05-07-2019 07:29 PM