Archive file using NiFI

[#1]

Using NiFi, I am processing a file. Once processing is done, I want to shift it to an archive folder. I want it to keep only say 30 days files and older files to be deleted. How to do. One more question is how to have timestamp name appended to file name. Like if file name is file1.txt then how to have it renamed like file1_09_09_2024_15_54.txt where we have 09_09_2024 as date and 15_54 as time(hour and mins)

[#2 · Accepted solution]

@AlokKumar 

You have multiple questions here, let me address them separately:

1. Renaming your content file: 
   
   1. A NiFi FlowFile holds its filename in a FlowFile Attribute named "filename".  You can use the UpdateAttribute processor to change/modify a FlowFiles filename through the use of the NiFi Expression Language (NEL)
   2. Add a dynamic new property to the UpdateAttribute processor using the "+"icon in upper right corner of [processors configure window. "Property" = FlowFile attribute to be modified (filename).  Value = A FlowFile NEL statement that manipulates the existing filename as desired.  Example: 

${filename:substringBeforeLast('.')}_${now():format('MM_dd_yyyy_hh_mm')}.${filename:substringAfterLast('.')}​

• With above example, I passed my UpdateAttribute a FlowFile with a "filename" attribute of "testfile.txt".  The above sample NEL statement modified the filename attribute on the FlowFile to "testfile_09_09_2024_12_57.txt".
  
  
• Archive a FlowFile: 
  1. You were not clear on how you want to handle your archived files.  Do you have some external service that monitors and cleans out your archive folder? NiFi has a PutFile processor that can write FlowFiles to some local directory on your local NiFi instance or other processors that may be able to write to an external service.
  2. If you are looking for NiFi to monitor that directory and purge archived files after 30 days, you would need to build another NiFi dataflow to handle that.  For example: GetFile (configured with minimum File age of 30 day) which you then auto-terminate its "success" relationship.  This will consume files older then 30 days and purge them from the configured directory.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3]

Hi @MattWho file naming part worked properly. Thanks

Could you please elaborare on the purging part where I want older files(older than 30 days) to be deleted.

[#4]

@AlokKumar 

The idea is that after renaming your FlowFile, you use the putFile processor to write that file to your archive directories on each of the NiFi instances.  That ends that dataflow.

You then start a new dataflow that consists of only the GetFile processor with its success relationship terminated and the minimum File Age property set to "30 days".  Start this processor and it will continuously check the target archive directory for any files where the last modified timestamp has exceeded 30 days.  Those files older then 30 days will be consumed and removed from archive assuming property "Keep Source File"  is set to "false".  Since the success "relationship" is set to auto-terminate, The FlowFile produced is terminated.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#5]

ok thank you