Support Questions
Find answers, ask questions, and share your expertise

Execute Gsutil from Nifi

Execute Gsutil from Nifi

Explorer

Hi Team,

we have observed that PUTGCSOBJECT processor in NiFi uploads the flowfile in one request which lowers down the upload speed for that file transfer .

we tried to use the GSUTIL utility to upload the same file with resumable uploads option and the speed was 10 times fasted as it performs multiple parallel uploads with the following command

gsutil -D -o GSUtil:parallel_composite_upload_threshold=100M cp <source file path> gs://<destination cloud bucket name>/

I tried with the ExecuteProcess processor to execute the above command but processor is not able to parse the command and landing into the parsing error.

Is some one aware about this performance or anybody has any alternative approach for nifi to upload the file in parts and in parallel and then perform compose operation as gsutils does ?