Update HDP to Spark 2.4.0

[#1]

Is there an update of HDP scheduled to support the new Spark 2.4.0?

[#2]

@Robo C

HDP ships with its own spark the current HDP 3.1 is bundled with Apache Spark 2.3.2. Following the merger with Cloudera, I doubt whether there will be another release soon like HDP 3.4 or HDP 3.5 but rather looking forward to a completely new integrated single offering of the best of both worlds (HW & CDH).

I would think now both companies are busy designing the integration roadmap !!. This is my personal option and NOT that of hortonworks or it's employee.

HTH

[#3]

Is there any way to update HDP to spark 2.4.0 manually then ?

[#4]

Hi @Geoffrey Shelton Okot,

I too have the same question but if we are upgrading Spark 2.4 manually does it supports HDP with all other dependencies?

Because as you have mentioned it has its own spark with version 2.3.2.

Regards,

Manjunath P N

[#5]

Maybe it is possible to install 2.4.0 manually and keep the 2.3.2 aside. This would not break anything but provide 2.4.0 if needed

[#6]

Since version 2.4.0 is a major update to Spark, it would be great for the company to release a (last?) update to HDP in order to include it.

[#7]

@Robo C @Manjunath P N

Unfortunately, you cannot independently upgrade an HDP component because Interoperability will be compromised. HWX advises against such practice because your subsequent major/minor upgrade will fail due to internal version checks during the upgrade.

If you found this answer addressed your question, please take a moment to log in and click the "accept" link on the answer. This will ensure other HCC users can quickly find the solution when they encounter the same issue.

[#8]

Geoffrey, thanks for your input. Nevertheless, since my original question is not about manual upgrade but about an official HDP update (exactly for the reasons you stated above), I would prefer someone from the company to reply. Any way we can get an official answer?

[#9]

@Robo C

Surely it's better you tag the HWX employees @Jay Kumar SenSharma @Aditya Sirna will react promptly

[#10]

@Jay Kumar SenSharma @Aditya Sirna