Why 2 versions of Hive and Spark in HDP 2.6.5

[#1]

I am planning to install HDP 2.6.5 and was checking the components and their versions. I noticed

• Apache Hive 1.2.1
• Apache Hive 2.1.0

and

• Apache Spark 1.6.3
• Apache Spark 2.3.0

Why are there two versions of the same component? Any specific reason?

[#2 · Accepted solution]

@Nanda Kumar

There are major changes between both hive and spark versions. So you need to decide which to install. And is possible to install both in same cluster.

For hive the differences between 1.2.1 and 2.1 are big, in case you like to take a look please check here:

https://hortonworks.com/blog/announcing-apache-hive-2-1-25x-faster-queries-much/

For spark 1.6.x and 2.x also we have major differences as well, you can read some more here:

https://hortonworks.com/blog/welcome-apache-spark-2-0/

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

[#3]

@Nanda Kumar Yes, both are compatible and supported! You can install both in the same cluster and use both of them. If you are satisfied with the answer please remember to login and mark the answer as "Accepted" Thanks!

[#4]

@Felix Albani , I'm currently on HDP 2.6.5 and would like to install or even preferably upgrade to Hive 2.1 . Do you happen to know any documentation on how to perform the install/upgrade? Unfortunately I can not see Hive 2.1 in the list of services that I can add.

[#5]

@Felix Albani. Thank you for the response. Are both the versions of both components compatible with the underlying components like Hadoop & HBase versions which are part of HDP 2.6.5 despite having huge changes between those versions?