Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

how to zip folder using nifi?

avatar
New Contributor

 

We have nifi flows like below
ListSFTP >> ... >> FetchSFTP >> ... >> MergeContent >> ... >> PutFTP

- FetchSFTP fetch all files in yyyymmdd folder( ex : ./A1/20191018 ) one by one
- MergeContent make all files into one zipfile.

It works well, but fetching all files one by one takes long time. also it needs lots of memory. Therefore we want to change above process into below


step1) Compress all files of one folder( ex : ./A1/20191018 --> 20191018.zip )
step2) Fetch this one zipfile
step3) PutFTP


In this process, I dont know about step1
How can I Compress all files in folder?

Thanks

 


All files are in Windows.
Nifi is in Linux.

 

1 ACCEPTED SOLUTION

avatar
Super Mentor

@maybegoodman 

 

NiFi is designed to operate on data which has been consumed by NiFi and placed in FlowFiles.

That being said, it sounds like what you are trying to do here is zip all the files on the target SFTP server before consuming the zipped file by NiFi.

 

Perhaps you could use the ExecuteStreamCommand processor to use ssh to manually zip the contents of the target SFTP server based on a configured scheduling cron.
Then have your listSFTP processor only list .zip file instead.

if the ExecuteStreamCommand is not adequate, there are several Script based processors perhaps you can use to write a custom script that handles step 1 fo connecting to SFTO server and zipping the files.

 

Hope this helps,

Matt

View solution in original post

2 REPLIES 2

avatar
Super Mentor

@maybegoodman 

 

NiFi is designed to operate on data which has been consumed by NiFi and placed in FlowFiles.

That being said, it sounds like what you are trying to do here is zip all the files on the target SFTP server before consuming the zipped file by NiFi.

 

Perhaps you could use the ExecuteStreamCommand processor to use ssh to manually zip the contents of the target SFTP server based on a configured scheduling cron.
Then have your listSFTP processor only list .zip file instead.

if the ExecuteStreamCommand is not adequate, there are several Script based processors perhaps you can use to write a custom script that handles step 1 fo connecting to SFTO server and zipping the files.

 

Hope this helps,

Matt

avatar
New Contributor

Thankyou MattWho

I  added Executestreamcommand processer before fetching data.