Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Can we upgrade Spark in HDP-2.3.2

avatar

HI,

Recently we install HDP-2.3.2.0-2950, in that we can see Spark-1.3.1.2.3

Can we upgrade Spark to 1.6 in same Hadoop version> And How?

1 ACCEPTED SOLUTION

avatar
Master Mentor

@Vishal Jain

I don't think you can upgrade spark version without upgrading to Latest version of HDP.

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_upgrading_hdp_manually/content/configurin...

See this http://hortonworks.com/hadoop-tutorial/a-lap-around-apache-spark/

HDP 2.4 is GA now and it comes with Spark 1.6

You can try the above link in your environment.

See this http://hortonworks.com/blog/announcing-the-general-availability-of-hortonworks-data-platform-2-4/

Starting with HDP 2.4, Hortonworks plans to release platform updates in two different cadences:

  • Core HDP services
  • Extended HDP services

For those customers whose requirements are met by Spark 1.5.2 or earlier, they can remain on HDP 2.3.x with the understanding that the same core exists between HDP 2.4 and HDP 2.3.4. There is no need to upgrade and if there happens to be any additional issues found on Hadoop 2.7.1, maintenance releases will be created to address these defects for both the HDP 2.3.x release and the HDP 2.4.x release – in an attempt to keep the core aligned and maintain stability.

View solution in original post

4 REPLIES 4

avatar
Master Mentor

@Vishal Jain

I don't think you can upgrade spark version without upgrading to Latest version of HDP.

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_upgrading_hdp_manually/content/configurin...

See this http://hortonworks.com/hadoop-tutorial/a-lap-around-apache-spark/

HDP 2.4 is GA now and it comes with Spark 1.6

You can try the above link in your environment.

See this http://hortonworks.com/blog/announcing-the-general-availability-of-hortonworks-data-platform-2-4/

Starting with HDP 2.4, Hortonworks plans to release platform updates in two different cadences:

  • Core HDP services
  • Extended HDP services

For those customers whose requirements are met by Spark 1.5.2 or earlier, they can remain on HDP 2.3.x with the understanding that the same core exists between HDP 2.4 and HDP 2.3.4. There is no need to upgrade and if there happens to be any additional issues found on Hadoop 2.7.1, maintenance releases will be created to address these defects for both the HDP 2.3.x release and the HDP 2.4.x release – in an attempt to keep the core aligned and maintain stability.

avatar

Thanks @Neeraj Sabharwal

Just want to chekc then can we go for Spark-1.5 on HDP-2.32 ? And what could be steps to upgrade.

avatar
Master Mentor

avatar
Master Mentor

Spark 1.6 is only supported in 2.4. There are no major changes in versions since 2.3.2, consider using latest HDP for latest spark