Apache Spark 2.3.2 and HDP 3.1

[#1]

hi all

after review on the new HDP version - 3.1

we saw that HDP 3.1 support only spark 2.3.2

since we must work with spark 2.4.0

my question - can we install the HDP 3.1 and upgrade the spark to 2.4.0 manually ?

[#2]

The above question was originally posted in the Community Help track. On Thu Jun 27 19:17 UTC 2019, a member of the HCC moderation staff moved it to the Data Science & Advanced Analytics track. The Community Help Track is intended for questions about using the HCC site itself, not technical installation questions about Spark or HDP.

[#3]

@Michael Bronson

That is correct that the HDP 3.1 is tested and certified with Spark 2.3.2

https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/comp_versions.html

Upgrading individual components manually may cause problems and can introduce issues later. Hence it is not recommended.

So at this point it is hard to provide any doc/reference where you can find Spark2.4 working with HDP 3.1 (as it does not comes under tested and certified version) Hence if you find any trouble in integrating that then you might run out of resources/references. Hence it will be best if you can stick to the tested and certified version.

[#4]

@Dear Jay , I will be with you honestly , this is a big problem that HDP latest version not include many of necessary latest version , for example - kafka version , on HDP 3.1 , if you compare the confluent kafka latest version with the kafka in HDP 3.1 , you can see the kafka version in HDP 3.1 is very old , so from my experience this could some users or companies not use the HDP kit and use a separate application from separate vendors

[#5]

@Dear jay - when yoiu think spark 2.4.0 will be part of HDP version ?