question how to zip folder using nifi? in Support Questions

how to zip folder using nifi?

maybegoodman — Fri, 18 Oct 2019 06:52:19 GMT

We have nifi flows like below
ListSFTP >> ... >> FetchSFTP >> ... >> MergeContent >> ... >> PutFTP

- FetchSFTP fetch all files in yyyymmdd folder( ex : ./A1/20191018 ) one by one
- MergeContent make all files into one zipfile.

It works well, but fetching all files one by one takes long time. also it needs lots of memory. Therefore we want to change above process into below

step1) Compress all files of one folder( ex : ./A1/20191018 --> 20191018.zip )
step2) Fetch this one zipfile
step3) PutFTP

In this process, I dont know about step1
How can I Compress all files in folder?

Thanks

All files are in Windows.
Nifi is in Linux.

Re: how to zip folder using nifi?

MattWho — Fri, 18 Oct 2019 13:13:51 GMT

@maybegoodman

NiFi is designed to operate on data which has been consumed by NiFi and placed in FlowFiles.

That being said, it sounds like what you are trying to do here is zip all the files on the target SFTP server before consuming the zipped file by NiFi.

Perhaps you could use the ExecuteStreamCommand processor to use ssh to manually zip the contents of the target SFTP server based on a configured scheduling cron.
Then have your listSFTP processor only list .zip file instead.

if the ExecuteStreamCommand is not adequate, there are several Script based processors perhaps you can use to write a custom script that handles step 1 fo connecting to SFTO server and zipping the files.

Hope this helps,

Matt

Re: how to zip folder using nifi?

maybegoodman — Mon, 28 Oct 2019 09:06:15 GMT

Thankyou MattWho

I added Executestreamcommand processer before fetching data.