Created 07-26-2024 06:26 AM
Hi,
If you search the internet you might find something that would help in your case. Please refer to this this: https://stackoverflow.com/questions/37530121/putfile-append-file
Here is my suggestion if you dont want to write\use some custom code.
Basically, if you think of the New Content (GenerateFlowFile) as the new data that you got , the idea is to do a Merge regardless, the trick is when you try to Fetch the file if it exist you will end up merging the content of the old to the new, and if it doesnt exist (not.found relationship) you will merge new content to empty string (using ReplaceText from not.found rel) . However there are few caveat with this design that you might need to address:
1- Since the Merge Content uses Text Strategy ( see config below) and it uses newline for Demarcator (delimiter). If the file doesnt exist (first time) it will add empty newline towards the end. you can solve this by using replace text or other methods so its not big deal.
2- The merge order is not predictable so you might get the new content at the top vs being appended to the bottom. That is because the merge content gets the New Content first before the old content. If the order is important , then you can use something like EnforceOrder Processor where you set an integer attribute for order priority on each content. If the order is not important then you can ignore.
3-You need to preserve the filename since MergeContent will produce new file name , however the MergeContent reads an attribute "segment.original.filename" if it finds it , it will use whatever filename specified there.
With that here are the different processor configurations:
1 - GenerateFlowFile (New Content):
- filename: simulate having file content with filename (you might have that already)
- segment.original.filename : this is used by MergeContent to set the correct filename after merge.
2- FetchFile:
Make sure to set the marked properties as such.
Also Important, under settings tab make sure to set the Penalty Duration to 0 sec. This is because for some reason when file is not found and the flowfile is directed to not.found relationship it will get penalized! not sure why even when I set the Log Level for Not Found to None.
3- ReplaceText (Replace Orig with Empty String):
4- Merge Content:
For the Demarcator (delimiter) use Shift+Enter for the newline character
5- PutFile:
Hope that helps. if it does please accept the solution and feel free to add your input in case you were able to make it work for your case so other can benefit.
Thanks