Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to update Spark to fix-version available in Hortonworks Maven repo

Highlighted

How to update Spark to fix-version available in Hortonworks Maven repo

New Contributor

While developing some code using Structured Streaming, I came across a code generation bug, which exists in Spark 2.1.1, but has been resolved in 2.2.0. Looking to make my code work on the current HDP, I saw that the latest Spark-release in the Maven repository was 2.1.1.2.6.1.9-1. That version no longer has the bug I was encountering in vanilla Spark 2.1.1.

After looking at our HDP installation, I realized, that the only Spark version currently being distributed is 2.1.1.2.6.1.0-129 which sadly still has this bug. I presume, that the bug has been fixed upon a customer request in on of the 9 hotfixes released since the 2.6.1 release. My question is - how would I distribute these hotfix-Spark releases using Ambari in a maintainable manner?

Obviously, I could manually download the spark-components from the repository and build a custom installation, but I assume that there's an easier way of accessing these artifacts via a package repository.