Need help on Spark Upgrade

[#1]

The spark versions are different in PROD and Staging environment and I have been asked to match.

In DEV it is of higher version :

[hdfs@admin0.exp spark]$ more RELEASE

Spark 1.5.2.2.3.4.0-3485 built for Hadoop 2.7.1.2.3.4.0-3485

[hdfs@admin0.exp spark]$ pwd

/usr/hdp/2.3.4.0-3485/spark

In PROD it is of lower version :

[hdfs@admin1.prod.iad spark]$ more RELEASE

Spark 1.3.1.2.3.0.0-2557 built for Hadoop 2.7.1.2.3.0.0-2557

[hdfs@admin1.prod.iad spark]$ pwd

/usr/hdp/2.3.0.0-2557/spark

If you see closely HDP version/release is also different : In DEV it is 2.3.4.0-3485, whereas in PROD it is 2.3.0.0-2557

Do you advise me to upgrade the HDP or Can I just upgrade just the spark ?

[#2 · Accepted solution]

@Kumar Veerappan

We recommend to upgrade HDP stack so that you can get appropriate version of spark which comes with stack instead of manually installing spark.

[#3]

Please start from below doc according to your current and target version.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.1.1.0/bk_upgrading_Ambari/content/_upgrading_hdp_...

[#4]

Hi @Kumar Veerappan please let me know if you need any other info on this or kindly accept the answer to close this thread.

[#5]

I am little confused..In Ambari the spark version is as follows :

Spark

1.5.2.2.3

Installed

Apache Spark is a fast and general engine for large-scale data processing.

[#6]

This is after upgrade? once you install it you can do cat RELEASE which will show you full version.

[#7]

Hi @Kumar Veerappan any further issue on this?

[#8]

@Jitendra Yadav This is even before the upgrade. I am little confused because in AMBARI the spark version says 1.5.2 whereas in side the node the version says 1.3.1

[#9]

@Kumar Veerappan did you finalized the upgrade?

[#10]

@Jitendra Yadav

Yes I have to upgrade the complete HDP stack from 2.3.0 to 2.3.4