My research indicates that, unlike those available for HDFS or S3, there does not appear to be a Google Cloud Storage (GCS) processor. As such, when reading and writing files to GCS it appears that the ExecuteScript is the processor to be used. Is there a GCS processor for NiFI?
There's no out of the box processor that I'm aware of. You could use ExecuteProcess if there is a CLI tool already doing the job. Maybe consider mounting a GCS volume as a local drive and access fir Put/GetFile. But no native processor today.
Thanks for providing your recommendation. Invoking GCloud CLI from within a Linux/Window shell script has worked well for me, in the past. Invoking these scripts from within an 'ExecuteProcess' processor in NiFi will be an solid improvement in approach.
However, since Google Cloud Fuse, which is used to mount a Google Cloud Storage (GCS) bucket as if it were a local directory/drive, will not become GA (https://cloud.google.com/storage/docs/gcs-fuse), though it will remain open source, the NiFi design you first recommended is preferred in my specific client's use case.
I'm considering creating a suite of NiFi processors for GCP (DeleteGCSObject; FetchGCSObject; etc.). Too bad the GCP market share is so small that if naturally falls into the back-end of the NiFi product development queue. Regardless, it will be a worthwhile experience to explore the NiFi Processor API along those lines.
Your idea to create NiFi processors for GCP sounds great. The amazon processors came about as a result of very similar community effort. I am quite sure the apache nifi community would help with such a contribution.