The spark versions are different in PROD and Staging environment and I have been asked to match.
In DEV it is of higher version :
[email@example.com spark]$ more RELEASE
Spark 18.104.22.168.3.4.0-3485 built for Hadoop 22.214.171.124.3.4.0-3485
[firstname.lastname@example.org spark]$ pwd
In PROD it is of lower version :
[email@example.com spark]$ more RELEASE
Spark 126.96.36.199.3.0.0-2557 built for Hadoop 188.8.131.52.3.0.0-2557
[firstname.lastname@example.org spark]$ pwd
If you see closely HDP version/release is also different : In DEV it is 184.108.40.206-3485, whereas in PROD it is 220.127.116.11-2557
Do you advise me to upgrade the HDP or Can I just upgrade just the spark ?
I am little confused..In Ambari the spark version is as follows :
|Spark||18.104.22.168.3||Installed||Apache Spark is a fast and general engine for large-scale data processing.|