Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi processor for Google Cloud Storage

NiFi processor for Google Cloud Storage

New Contributor

My research indicates that, unlike those available for HDFS or S3, there does not appear to be a Google Cloud Storage (GCS) processor. As such, when reading and writing files to GCS it appears that the ExecuteScript is the processor to be used. Is there a GCS processor for NiFI?

4 REPLIES 4

Re: NiFi processor for Google Cloud Storage

There's no out of the box processor that I'm aware of. You could use ExecuteProcess if there is a CLI tool already doing the job. Maybe consider mounting a GCS volume as a local drive and access fir Put/GetFile. But no native processor today.

Re: NiFi processor for Google Cloud Storage

New Contributor

Thanks for providing your recommendation. Invoking GCloud CLI from within a Linux/Window shell script has worked well for me, in the past. Invoking these scripts from within an 'ExecuteProcess' processor in NiFi will be an solid improvement in approach.

However, since Google Cloud Fuse, which is used to mount a Google Cloud Storage (GCS) bucket as if it were a local directory/drive, will not become GA (https://cloud.google.com/storage/docs/gcs-fuse), though it will remain open source, the NiFi design you first recommended is preferred in my specific client's use case.

I'm considering creating a suite of NiFi processors for GCP (DeleteGCSObject; FetchGCSObject; etc.). Too bad the GCP market share is so small that if naturally falls into the back-end of the NiFi product development queue. Regardless, it will be a worthwhile experience to explore the NiFi Processor API along those lines.

Cheers

@andrewgrande

Re: NiFi processor for Google Cloud Storage

@Charles Clifford

Your idea to create NiFi processors for GCP sounds great. The amazon processors came about as a result of very similar community effort. I am quite sure the apache nifi community would help with such a contribution.

Re: NiFi processor for Google Cloud Storage

New Contributor