Can we upgrade Spark in HDP-2.3.2

[#1]

HI,

Recently we install HDP-2.3.2.0-2950, in that we can see Spark-1.3.1.2.3

Can we upgrade Spark to 1.6 in same Hadoop version> And How?

[#2 · Accepted solution]

@Vishal Jain

I don't think you can upgrade spark version without upgrading to Latest version of HDP.

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_upgrading_hdp_manually/content/configurin...

See this http://hortonworks.com/hadoop-tutorial/a-lap-around-apache-spark/

HDP 2.4 is GA now and it comes with Spark 1.6

You can try the above link in your environment.

See this http://hortonworks.com/blog/announcing-the-general-availability-of-hortonworks-data-platform-2-4/

Starting with HDP 2.4, Hortonworks plans to release platform updates in two different cadences:

• Core HDP services
• Extended HDP services

For those customers whose requirements are met by Spark 1.5.2 or earlier, they can remain on HDP 2.3.x with the understanding that the same core exists between HDP 2.4 and HDP 2.3.4. There is no need to upgrade and if there happens to be any additional issues found on Hadoop 2.7.1, maintenance releases will be created to address these defects for both the HDP 2.3.x release and the HDP 2.4.x release – in an attempt to keep the core aligned and maintain stability.

[#3]

Thanks @Neeraj Sabharwal

Just want to chekc then can we go for Spark-1.5 on HDP-2.32 ? And what could be steps to upgrade.

[#4]

@Vishal Jain

See this

https://community.hortonworks.com/questions/17175/how-to-upgrade-hdp-spark-version.html

Answer#2 is the best answer...

[#5]

Spark 1.6 is only supported in 2.4. There are no major changes in versions since 2.3.2, consider using latest HDP for latest spark