Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Need help on Spark Upgrade

avatar
Expert Contributor

The spark versions are different in PROD and Staging environment and I have been asked to match.

In DEV it is of higher version :

[hdfs@admin0.exp spark]$ more RELEASE

Spark 1.5.2.2.3.4.0-3485 built for Hadoop 2.7.1.2.3.4.0-3485

[hdfs@admin0.exp spark]$ pwd

/usr/hdp/2.3.4.0-3485/spark

In PROD it is of lower version :

[hdfs@admin1.prod.iad spark]$ more RELEASE

Spark 1.3.1.2.3.0.0-2557 built for Hadoop 2.7.1.2.3.0.0-2557

[hdfs@admin1.prod.iad spark]$ pwd

/usr/hdp/2.3.0.0-2557/spark

If you see closely HDP version/release is also different : In DEV it is 2.3.4.0-3485, whereas in PROD it is 2.3.0.0-2557

Do you advise me to upgrade the HDP or Can I just upgrade just the spark ?

1 ACCEPTED SOLUTION

avatar
Super Guru

@Kumar Veerappan

We recommend to upgrade HDP stack so that you can get appropriate version of spark which comes with stack instead of manually installing spark.

View solution in original post

12 REPLIES 12

avatar
Super Guru

@Kumar Veerappan

We recommend to upgrade HDP stack so that you can get appropriate version of spark which comes with stack instead of manually installing spark.

avatar
Super Guru

Please start from below doc according to your current and target version.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.1.1.0/bk_upgrading_Ambari/content/_upgrading_hdp_...

avatar
Super Guru

Hi @Kumar Veerappan please let me know if you need any other info on this or kindly accept the answer to close this thread.

avatar
Expert Contributor

I am little confused..In Ambari the spark version is as follows :

Spark1.5.2.2.3InstalledApache Spark is a fast and general engine for large-scale data processing.

avatar
Super Guru

This is after upgrade? once you install it you can do cat RELEASE which will show you full version.

avatar
Super Guru

Hi @Kumar Veerappan any further issue on this?

avatar
Expert Contributor

@Jitendra Yadav This is even before the upgrade. I am little confused because in AMBARI the spark version says 1.5.2 whereas in side the node the version says 1.3.1

avatar
Super Guru

@Kumar Veerappan did you finalized the upgrade?

avatar
Expert Contributor
@Jitendra Yadav

Yes I have to upgrade the complete HDP stack from 2.3.0 to 2.3.4