HDP Spark Connector for Google Cloud

[#1]

We are in process to migrate our HDP clusters to Google Cloud. Spark is heavily used by our applications and data scientists. GCS will be also heavily used to fetch raw data and post processed data. One challenge we have is the lack of support for Google Connector and capabilities around GCS and running Spark jobs efficiently on GCS stored data. This is not part of the HDP bundle and there is no information on whether this will be bundled and supported in the next versions. Does Hortonworks have a plan to include it in the following versions, also to support?

[#2 · Accepted solution]

@Jane Becker

True. The connector is not currently bundled or supported. I installed it manually and my preliminary tests were successful when using it with Spark, but I did not anything complicated or at scale. I checked recently with Engineering and there is a good chance that it will be supported in the second part of 2018. As this connector gets more attention and importance from the users community, its priority will increase and there will be a better chance that it will be supported sooner. As you may know, this connector does not seem supported even by Google.