Update HDP to Spark 2.4.0

[#1]

@asirna @jsensharma could you give some hint on how to upgrade to spark 2.4.0 in HDP 2.6.5 or HDP 3 ?

[#2]

@jsensharma @asirna any guidance on how to upgrade spark in hdp 2.6 and hdp 3 ?

In particular how to compile spark from source and then deploy it?

[#3]

If anyone has done manual Integration from Spark 2.3.1 to Spark 2.4.2 please share experience and steps.

Thanks