question Re: Can we upgrade Spark in HDP-2.3.2 in Archives of Support Questions (Read Only)

Can we upgrade Spark in HDP-2.3.2

Vishaljain000 — Wed, 02 Mar 2016 17:40:01 GMT

HI,

Recently we install HDP-2.3.2.0-2950, in that we can see Spark-1.3.1.2.3

Can we upgrade Spark to 1.6 in same Hadoop version> And How?

Re: Can we upgrade Spark in HDP-2.3.2

nsabharwal — Wed, 02 Mar 2016 17:49:06 GMT

@Vishal Jain

I don't think you can upgrade spark version without upgrading to Latest version of HDP.

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_upgrading_hdp_manually/content/configuring_and_upgrading_spark.html

See this http://hortonworks.com/hadoop-tutorial/a-lap-around-apache-spark/

HDP 2.4 is GA now and it comes with Spark 1.6

You can try the above link in your environment.

See this http://hortonworks.com/blog/announcing-the-general-availability-of-hortonworks-data-platform-2-4/

Starting with HDP 2.4, Hortonworks plans to release platform updates in two different cadences:

Core HDP services
Extended HDP services

For those customers whose requirements are met by Spark 1.5.2 or earlier, they can remain on HDP 2.3.x with the understanding that the same core exists between HDP 2.4 and HDP 2.3.4. There is no need to upgrade and if there happens to be any additional issues found on Hadoop 2.7.1, maintenance releases will be created to address these defects for both the HDP 2.3.x release and the HDP 2.4.x release – in an attempt to keep the core aligned and maintain stability.

Re: Can we upgrade Spark in HDP-2.3.2

aervits — Wed, 02 Mar 2016 17:56:07 GMT

Spark 1.6 is only supported in 2.4. There are no major changes in versions since 2.3.2, consider using latest HDP for latest spark

Re: Can we upgrade Spark in HDP-2.3.2

Vishaljain000 — Wed, 02 Mar 2016 18:08:54 GMT

Thanks @Neeraj Sabharwal

Just want to chekc then can we go for Spark-1.5 on HDP-2.32 ? And what could be steps to upgrade.

Re: Can we upgrade Spark in HDP-2.3.2

nsabharwal — Wed, 02 Mar 2016 19:32:36 GMT

@Vishal Jain

See this

https://community.hortonworks.com/questions/17175/how-to-upgrade-hdp-spark-version.html

Answer#2 is the best answer...