Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NiFi: Merge files based on attribute and send email notification

avatar
Contributor

I have a requirement to notify once the files on the destination is processed. 

  1. The pipeline is as follows: ListSFTP --> Update attribute (adds specific attribute for each state/province) ---> Send to Cloud Storage. 
  2. Each state has a different count of files. 
  3. Once the files are sent to cloud storage, notify the end user using email for each state. 

I have tried merging files based on a common attribute but, I need single files for each state so that I can send one notification for each state. 

 

1 ACCEPTED SOLUTION

avatar
Super Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
4 REPLIES 4

avatar
Super Mentor

@Jagapriyan 

Your described flow above does not mention the mergeContent processor which is what would be needed to merge multiple FlowFiles with matching attributes values into 1 output FlowFile.

Share your MergeContent processor configuration.

Additionally the ListSFTP processor does not download the content of the files form the remote server.  It is only used to list the files on the remote server and set attributes on the FlowFile that would be used by the FetchSFTP processor to actually download the content. 

How do you know when you have all the files for a given state?  Is this a continues feed of files?  Is this a daily job?  While file count is different per state, is count same per state?  What is the highest count and lowest count?

Thanks,
Matt

avatar
Contributor

Hi @MattWho 

Thanks for the time to respond. 

How do you know when you have all the files for a given state?  Usually, the files are status reports for the previous report. They are generated between 8am to 9am everyday. 

For example: If I have 2 countries US and Canada, there will be 90 files for US and 100 files for Canada. (the number of files may vary based on usage the date before)

The flow: We have a list SFTP processor and Fetch SFTP processor that are collecting these files and storing on GCP.

The requirement:  I need to send an email notifying that files are delivered. 

What I Tried:  Since I need to send one email for each country, I tried  "replacetext" processor to remove the contents of the flow file and then merge the files into one. Since I can't use a hardcoded number, I couldn't give a value in minimum entries. 

My current Merge content processor

Jagapriyan_0-1665664863474.png

 

 

avatar
Super Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Contributor

Thank you @Matt