Need help on Spark Upgrade

[#1]

The spark versions are different in PROD and Staging environment and I have been asked to match.

In DEV it is of higher version :

[hdfs@admin0.exp spark]$ more RELEASE

Spark 1.5.2.2.3.4.0-3485 built for Hadoop 2.7.1.2.3.4.0-3485

[hdfs@admin0.exp spark]$ pwd

/usr/hdp/2.3.4.0-3485/spark

In PROD it is of lower version :

[hdfs@admin1.prod.iad spark]$ more RELEASE

Spark 1.3.1.2.3.0.0-2557 built for Hadoop 2.7.1.2.3.0.0-2557

[hdfs@admin1.prod.iad spark]$ pwd

/usr/hdp/2.3.0.0-2557/spark

If you see closely HDP version/release is also different : In DEV it is 2.3.4.0-3485, whereas in PROD it is 2.3.0.0-2557

Do you advise me to upgrade the HDP or Can I just upgrade just the spark ?

[#2]

once upgrade completed we also need to clicked on finalize button to complete the upgrade process. if you have done this and still see previous version of spark then you should login to node and manually check spark package version through "rpm -qa|grep spark" command.

[#3]

@Jitendra Yadav

..thanks for your suggestion. Me being a newbie to this system : Can I upgrade through Ambari ?

[#4]

Agreed, you should at least upgrade the lower HDP version (...2.3.0...) to the newer HDP version (2.3.4.0-3485). It is best to get the default Spark version from the HDP install. Please see Table 1.1 at this link which describes the version associations for HDP, Ambari, and Spark:

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_spark-guide/content/ch_introduction-spark...