- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Need help on Spark Upgrade
- Labels:
-
Apache Hadoop
-
Apache Spark
Created 06-22-2016 08:34 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The spark versions are different in PROD and Staging environment and I have been asked to match.
In DEV it is of higher version :
[hdfs@admin0.exp spark]$ more RELEASE
Spark 1.5.2.2.3.4.0-3485 built for Hadoop 2.7.1.2.3.4.0-3485
[hdfs@admin0.exp spark]$ pwd
/usr/hdp/2.3.4.0-3485/spark
In PROD it is of lower version :
[hdfs@admin1.prod.iad spark]$ more RELEASE
Spark 1.3.1.2.3.0.0-2557 built for Hadoop 2.7.1.2.3.0.0-2557
[hdfs@admin1.prod.iad spark]$ pwd
/usr/hdp/2.3.0.0-2557/spark
If you see closely HDP version/release is also different : In DEV it is 2.3.4.0-3485, whereas in PROD it is 2.3.0.0-2557
Do you advise me to upgrade the HDP or Can I just upgrade just the spark ?
Created 06-22-2016 09:01 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We recommend to upgrade HDP stack so that you can get appropriate version of spark which comes with stack instead of manually installing spark.
Created 06-22-2016 09:01 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We recommend to upgrade HDP stack so that you can get appropriate version of spark which comes with stack instead of manually installing spark.
Created 06-23-2016 02:29 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Please start from below doc according to your current and target version.
Created 06-27-2016 07:02 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Kumar Veerappan please let me know if you need any other info on this or kindly accept the answer to close this thread.
Created 06-27-2016 09:27 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am little confused..In Ambari the spark version is as follows :
Spark | 1.5.2.2.3 | Installed | Apache Spark is a fast and general engine for large-scale data processing. |
Created 06-27-2016 09:31 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This is after upgrade? once you install it you can do cat RELEASE which will show you full version.
Created 06-29-2016 10:57 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Kumar Veerappan any further issue on this?
Created 06-29-2016 09:16 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Jitendra Yadav This is even before the upgrade. I am little confused because in AMBARI the spark version says 1.5.2 whereas in side the node the version says 1.3.1
Created 06-29-2016 09:17 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Kumar Veerappan did you finalized the upgrade?
Created 06-29-2016 09:19 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes I have to upgrade the complete HDP stack from 2.3.0 to 2.3.4