how to zip folder using nifi?

[#1]

We have nifi flows like below
ListSFTP >> ... >> FetchSFTP >> ... >> MergeContent >> ... >> PutFTP

- FetchSFTP fetch all files in yyyymmdd folder( ex : ./A1/20191018 ) one by one
- MergeContent make all files into one zipfile.

It works well, but fetching all files one by one takes long time. also it needs lots of memory. Therefore we want to change above process into below

step1) Compress all files of one folder( ex : ./A1/20191018 --> 20191018.zip )
step2) Fetch this one zipfile
step3) PutFTP

In this process, I dont know about step1
How can I Compress all files in folder?

Thanks

All files are in Windows.
Nifi is in Linux.

[#2 · Accepted solution]

@maybegoodman 

NiFi is designed to operate on data which has been consumed by NiFi and placed in FlowFiles.

That being said, it sounds like what you are trying to do here is zip all the files on the target SFTP server before consuming the zipped file by NiFi.

Perhaps you could use the ExecuteStreamCommand processor to use ssh to manually zip the contents of the target SFTP server based on a configured scheduling cron.
Then have your listSFTP processor only list .zip file instead.

if the ExecuteStreamCommand is not adequate, there are several Script based processors perhaps you can use to write a custom script that handles step 1 fo connecting to SFTO server and zipping the files.

Hope this helps,

Matt

[#3]

Thankyou MattWho

I  added Executestreamcommand processer before fetching data.