Without having to do HDP full-stack upgrade, need to manually upgrade spark from 2.3 to 2.4
Spark 2.4 is not officially tested, therefore it is not supported in HDP 2.6.5. If you still want to proceed with upgrading Spark version, you could find the procedure online.
Here is a list of supported component versions in HDP 2.6.5:
Actually, I did install spark 2.4 and set path on the Namenode2 where original Spark2.3 was residing - I can enter spark shell and verify it is 2.4, However, in Ambari Dashboard Services it still shows 2.3. Can you guide me to a weblink that can help?