We have a problem with our hortonworks installation, specifically with our spark version 1.6.
We have a streaming spark app which one fail with this error:
I know, obviously is resolved and if I download the most recent version of spark 1.6 I resolve the problem, but in my case I have the Hortonworks 2.5 version and I can not delete my current instalation and donwnload the new installation, actually I dont know if I can delete my current instalation of spark 1.6 of hortonworks and re-install again with the last version of spark because I think if do that I reinstall the same version with the same bug.
Anyway in the github repository they explain I should apply the patch of github, but here is my question.
How can I apply this patch in my installation and recompile all spark installation? some how to or step to step.............. or maybe Can I do that with ambari?
Thanks in advance.
The error you are referencing (SPARK-12617) seems to be resolved in Spark 1.6.1. HDP 2.5.0 has Spark 1.6.2 where this issue should be resolved. Also, you can have newer Spark version (Spark 2.2) if you upgrade to HDP 2.6.
If you really need to patch, work with your Hortonworks support.
Thanks a lot and I am sorry by the wrong information, actually we have HDP 2.4.2 and we have currently the spark 1.6.0 version. Please, Could you tell me from what version of HDP the version of spark has resolved the bug which one I told you? Maybe in the 2.4.3 version?
HDP 2.4.2 has Spark 1.6.1 and not 1.6.0 : https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_HDP_RelNotes/content/ch_relnotes_v242.ht...
So the issue should be resolved in your version. Are you sure you are hitting this bug ?
I am sorry again, I am not the admin of the system and the admin gave me wrong informartion. Confirmated, we have HDP 2.4.0 and we will update to HDP 2.4.2 where we will have the spark 1.6.2 and I hope we resolve rhe problem.
Thanks a lot and I am sorry by the missunderstood.
We have huge problem because we have updated to HDP 2.4.3 with spark 1.6.2 and we continue with the same bug about the library. Ho can We resolve this problem?