The spark versions are different in PROD and Staging environment and I have been asked to match.
In DEV it is of higher version :
[email@example.com spark]$ more RELEASE
Spark 126.96.36.199.3.4.0-3485 built for Hadoop 188.8.131.52.3.4.0-3485
[firstname.lastname@example.org spark]$ pwd
In PROD it is of lower version :
[email@example.com spark]$ more RELEASE
Spark 184.108.40.206.3.0.0-2557 built for Hadoop 220.127.116.11.3.0.0-2557
[firstname.lastname@example.org spark]$ pwd
If you see closely HDP version/release is also different : In DEV it is 18.104.22.168-3485, whereas in PROD it is 22.214.171.124-2557
Do you advise me to upgrade the HDP or Can I just upgrade just the spark ?
once upgrade completed we also need to clicked on finalize button to complete the upgrade process. if you have done this and still see previous version of spark then you should login to node and manually check spark package version through "rpm -qa|grep spark" command.
Agreed, you should at least upgrade the lower HDP version (...2.3.0...) to the newer HDP version (126.96.36.199-3485). It is best to get the default Spark version from the HDP install. Please see Table 1.1 at this link which describes the version associations for HDP, Ambari, and Spark: