Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to zip folder using nifi?

Solved Go to solution
Highlighted

how to zip folder using nifi?

New Contributor

 

We have nifi flows like below
ListSFTP >> ... >> FetchSFTP >> ... >> MergeContent >> ... >> PutFTP

- FetchSFTP fetch all files in yyyymmdd folder( ex : ./A1/20191018 ) one by one
- MergeContent make all files into one zipfile.

It works well, but fetching all files one by one takes long time. also it needs lots of memory. Therefore we want to change above process into below


step1) Compress all files of one folder( ex : ./A1/20191018 --> 20191018.zip )
step2) Fetch this one zipfile
step3) PutFTP


In this process, I dont know about step1
How can I Compress all files in folder?

Thanks

 


All files are in Windows.
Nifi is in Linux.

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: how to zip folder using nifi?

Master Guru

@maybegoodman 

 

NiFi is designed to operate on data which has been consumed by NiFi and placed in FlowFiles.

That being said, it sounds like what you are trying to do here is zip all the files on the target SFTP server before consuming the zipped file by NiFi.

 

Perhaps you could use the ExecuteStreamCommand processor to use ssh to manually zip the contents of the target SFTP server based on a configured scheduling cron.
Then have your listSFTP processor only list .zip file instead.

if the ExecuteStreamCommand is not adequate, there are several Script based processors perhaps you can use to write a custom script that handles step 1 fo connecting to SFTO server and zipping the files.

 

Hope this helps,

Matt

2 REPLIES 2

Re: how to zip folder using nifi?

Master Guru

@maybegoodman 

 

NiFi is designed to operate on data which has been consumed by NiFi and placed in FlowFiles.

That being said, it sounds like what you are trying to do here is zip all the files on the target SFTP server before consuming the zipped file by NiFi.

 

Perhaps you could use the ExecuteStreamCommand processor to use ssh to manually zip the contents of the target SFTP server based on a configured scheduling cron.
Then have your listSFTP processor only list .zip file instead.

if the ExecuteStreamCommand is not adequate, there are several Script based processors perhaps you can use to write a custom script that handles step 1 fo connecting to SFTO server and zipping the files.

 

Hope this helps,

Matt

Re: how to zip folder using nifi?

New Contributor

Thankyou MattWho

I  added Executestreamcommand processer before fetching data.

Don't have an account?
Coming from Hortonworks? Activate your account here