question Re: Install two versions of Spark on same cluster. in Archives of Support Questions (Read Only)

Install two versions of Spark on same cluster.

bandarusridhar1 — Wed, 12 Oct 2016 04:12:38 GMT

Hi,

Is it possible to install two version of Spark on the same cluster? I know that in HDP 2.5 it supports Spark 1.6 and Spark 2.0 but I want them on either HDP 2.3.6 to install Spark 1.5 and Spark 1.6.

The reason for above requirement is we are integrating the external tool with Hadoop and Spark, where one tool support only Spark 1.5 and other tool require Spark 1.6.

Any help is highly appricated.

Re: Install two versions of Spark on same cluster.

sunile_manjee — Wed, 12 Oct 2016 04:15:00 GMT

@SBandaru from a technical sense you could do it but this would not be supported. If spark 1.6.x is required I recommend upgrading to HDP 2.4.2. Regarding your tool which support older version, I would reach out to vendor and ask them to come up to latest release. Spark has moved on to 2.0. Being on 1.5 well behind the ball in terms of spark.

Re: Install two versions of Spark on same cluster.

cstanca — Sat, 15 Oct 2016 10:32:56 GMT

@SBandaru

Yes. Actually, it is technically possible and even done. HDP 2.5 includes two versions of Spark: 1.6.2 production level and 2.0 technical preview. They co-exist having different timeline server. You can add Spark 2.0 using Ambari UI and "Add Service". In this case, the reason is to provide a preview of Spark 2.0, however, it is a business decision whether it makes sense.

If any of the responses was helpful, don't forget to vote/accept best answer.

Re: Install two versions of Spark on same cluster.

xfox — Sat, 15 Oct 2016 20:02:38 GMT

I have cluster with two spark versions, one was installed with cluster, one manually for zeppelin and livy server, no issues.

Re: Install two versions of Spark on same cluster.

prnuamat — Thu, 15 Jun 2017 23:52:52 GMT

I have installed Spark 2.1 manually on HDP 2.3.4 while there is another version Spark 1.5 already installed via HDP. When i try to run jobs in yarn cluster mode spark 2.1 is resolving to HDP 2.3.4 spark libraries and resulting in bad substitution errors. Any ideas how you were able to resolve this when using two spark versions ?

Re: Install two versions of Spark on same cluster.

prnuamat — Fri, 16 Jun 2017 00:30:31 GMT