Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Update HDP to Spark 2.4.0

Highlighted

Update HDP to Spark 2.4.0

New Contributor

Is there an update of HDP scheduled to support the new Spark 2.4.0?

12 REPLIES 12

Re: Update HDP to Spark 2.4.0

Mentor

@Robo C

HDP ships with its own spark the current HDP 3.1 is bundled with Apache Spark 2.3.2. Following the merger with Cloudera, I doubt whether there will be another release soon like HDP 3.4 or HDP 3.5 but rather looking forward to a completely new integrated single offering of the best of both worlds (HW & CDH).

I would think now both companies are busy designing the integration roadmap !!. This is my personal option and NOT that of hortonworks or it's employee.

HTH

Re: Update HDP to Spark 2.4.0

New Contributor

Is there any way to update HDP to spark 2.4.0 manually then ?

Re: Update HDP to Spark 2.4.0

New Contributor

Hi @Geoffrey Shelton Okot,

I too have the same question but if we are upgrading Spark 2.4 manually does it supports HDP with all other dependencies?

Because as you have mentioned it has its own spark with version 2.3.2.

Regards,

Manjunath P N

Re: Update HDP to Spark 2.4.0

New Contributor

Maybe it is possible to install 2.4.0 manually and keep the 2.3.2 aside. This would not break anything but provide 2.4.0 if needed

Re: Update HDP to Spark 2.4.0

New Contributor

Since version 2.4.0 is a major update to Spark, it would be great for the company to release a (last?) update to HDP in order to include it.

Re: Update HDP to Spark 2.4.0

Mentor

@Robo C @Manjunath P N

Unfortunately, you cannot independently upgrade an HDP component because Interoperability will be compromised. HWX advises against such practice because your subsequent major/minor upgrade will fail due to internal version checks during the upgrade.

If you found this answer addressed your question, please take a moment to log in and click the "accept" link on the answer. This will ensure other HCC users can quickly find the solution when they encounter the same issue.

Re: Update HDP to Spark 2.4.0

New Contributor

Geoffrey, thanks for your input. Nevertheless, since my original question is not about manual upgrade but about an official HDP update (exactly for the reasons you stated above), I would prefer someone from the company to reply. Any way we can get an official answer?

Re: Update HDP to Spark 2.4.0

Mentor

@Robo C

Surely it's better you tag the HWX employees @Jay Kumar SenSharma @Aditya Sirna will react promptly


Re: Update HDP to Spark 2.4.0

New Contributor