Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

apply patch in spark 1.6 installation in HDP 2.5

Highlighted

apply patch in spark 1.6 installation in HDP 2.5

New Contributor

We have a problem with our hortonworks installation, specifically with our spark version 1.6.

We have a streaming spark app which one fail with this error:

https://github.com/apache/spark/pull/10621

I know, obviously is resolved and if I download the most recent version of spark 1.6 I resolve the problem, but in my case I have the Hortonworks 2.5 version and I can not delete my current instalation and donwnload the new installation, actually I dont know if I can delete my current instalation of spark 1.6 of hortonworks and re-install again with the last version of spark because I think if do that I reinstall the same version with the same bug.

Anyway in the github repository they explain I should apply the patch of github, but here is my question.

How can I apply this patch in my installation and recompile all spark installation? some how to or step to step.............. or maybe Can I do that with ambari?

Thanks in advance.

5 REPLIES 5

Re: apply patch in spark 1.6 installation in HDP 2.5

Hi @carlos palomares

The error you are referencing (SPARK-12617) seems to be resolved in Spark 1.6.1. HDP 2.5.0 has Spark 1.6.2 where this issue should be resolved. Also, you can have newer Spark version (Spark 2.2) if you upgrade to HDP 2.6.

If you really need to patch, work with your Hortonworks support.

Thanks

Highlighted

Re: apply patch in spark 1.6 installation in HDP 2.5

New Contributor

Hi,

Thanks a lot and I am sorry by the wrong information, actually we have HDP 2.4.2 and we have currently the spark 1.6.0 version. Please, Could you tell me from what version of HDP the version of spark has resolved the bug which one I told you? Maybe in the 2.4.3 version?

Thanks.

Regards.

Highlighted

Re: apply patch in spark 1.6 installation in HDP 2.5

Hi @carlos palomares

HDP 2.4.2 has Spark 1.6.1 and not 1.6.0 : https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_HDP_RelNotes/content/ch_relnotes_v242.ht...

So the issue should be resolved in your version. Are you sure you are hitting this bug ?

Highlighted

Re: apply patch in spark 1.6 installation in HDP 2.5

New Contributor

I am sorry again, I am not the admin of the system and the admin gave me wrong informartion. Confirmated, we have HDP 2.4.0 and we will update to HDP 2.4.2 where we will have the spark 1.6.2 and I hope we resolve rhe problem.

Thanks a lot and I am sorry by the missunderstood.

Highlighted

Re: apply patch in spark 1.6 installation in HDP 2.5

New Contributor

Hi Again,

We have huge problem because we have updated to HDP 2.4.3 with spark 1.6.2 and we continue with the same bug about the library. Ho can We resolve this problem?

Please Help.

Don't have an account?
Coming from Hortonworks? Activate your account here