question Re: Need help on Spark Upgrade in Support Questions

Need help on Spark Upgrade

kums — Thu, 23 Jun 2016 03:34:11 GMT

The spark versions are different in PROD and Staging environment and I have been asked to match.

In DEV it is of higher version :

[hdfs@admin0.exp spark]$ more RELEASE

Spark 1.5.2.2.3.4.0-3485 built for Hadoop 2.7.1.2.3.4.0-3485

[hdfs@admin0.exp spark]$ pwd

/usr/hdp/2.3.4.0-3485/spark

In PROD it is of lower version :

[hdfs@admin1.prod.iad spark]$ more RELEASE

Spark 1.3.1.2.3.0.0-2557 built for Hadoop 2.7.1.2.3.0.0-2557

[hdfs@admin1.prod.iad spark]$ pwd

/usr/hdp/2.3.0.0-2557/spark

If you see closely HDP version/release is also different : In DEV it is 2.3.4.0-3485, whereas in PROD it is 2.3.0.0-2557

Do you advise me to upgrade the HDP or Can I just upgrade just the spark ?

Re: Need help on Spark Upgrade

jyadav — Thu, 23 Jun 2016 04:01:00 GMT

@Kumar Veerappan

We recommend to upgrade HDP stack so that you can get appropriate version of spark which comes with stack instead of manually installing spark.

Re: Need help on Spark Upgrade

kums — Thu, 23 Jun 2016 21:21:47 GMT

@Jitendra Yadav

..thanks for your suggestion. Me being a newbie to this system : Can I upgrade through Ambari ?

Re: Need help on Spark Upgrade

jyadav — Thu, 23 Jun 2016 21:29:48 GMT

Please start from below doc according to your current and target version.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.1.1.0/bk_upgrading_Ambari/content/_upgrading_hdp_stack.html

Re: Need help on Spark Upgrade

phargis — Fri, 24 Jun 2016 01:31:22 GMT

Agreed, you should at least upgrade the lower HDP version (...2.3.0...) to the newer HDP version (2.3.4.0-3485). It is best to get the default Spark version from the HDP install. Please see Table 1.1 at this link which describes the version associations for HDP, Ambari, and Spark:

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_spark-guide/content/ch_introduction-spark.html

Re: Need help on Spark Upgrade

jyadav — Mon, 27 Jun 2016 14:02:45 GMT

Hi @Kumar Veerappan please let me know if you need any other info on this or kindly accept the answer to close this thread.

Re: Need help on Spark Upgrade

kums — Tue, 28 Jun 2016 04:27:39 GMT

I am little confused..In Ambari the spark version is as follows :

Spark

1.5.2.2.3

Installed

Apache Spark is a fast and general engine for large-scale data processing.

Re: Need help on Spark Upgrade

jyadav — Tue, 28 Jun 2016 04:31:08 GMT

This is after upgrade? once you install it you can do cat RELEASE which will show you full version.

Re: Need help on Spark Upgrade

jyadav — Wed, 29 Jun 2016 17:57:16 GMT

Hi @Kumar Veerappan any further issue on this?

Re: Need help on Spark Upgrade

kums — Thu, 30 Jun 2016 04:16:49 GMT

@Jitendra Yadav This is even before the upgrade. I am little confused because in AMBARI the spark version says 1.5.2 whereas in side the node the version says 1.3.1

Re: Need help on Spark Upgrade

jyadav — Thu, 30 Jun 2016 04:17:56 GMT

@Kumar Veerappan did you finalized the upgrade?

Re: Need help on Spark Upgrade

kums — Thu, 30 Jun 2016 04:19:48 GMT

@Jitendra Yadav

Yes I have to upgrade the complete HDP stack from 2.3.0 to 2.3.4

Re: Need help on Spark Upgrade

jyadav — Thu, 30 Jun 2016 05:01:51 GMT

once upgrade completed we also need to clicked on finalize button to complete the upgrade process. if you have done this and still see previous version of spark then you should login to node and manually check spark package version through "rpm -qa|grep spark" command.