Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

file watcher and email notification in NIFI.

Highlighted

file watcher and email notification in NIFI.

New Contributor

Hello,

In file watcher scenario, everyday i am getting new and updated file in the local folder.

The incoming file count is varies in everyday.

Need to send email notification with updated and new file name list with specific schedule time.

Using Listfile processor, I am getting all new and updated file name.

How to list out all filename in one file and send that file using email notification.

Thanks

3 REPLIES 3

Re: file watcher and email notification in NIFI.

Master Guru

@Mitthu Wagh

-

Just so I understand, can you share complete use case?

-

Are you listing and fetching the new files or just listing them?
Are you running this listing using a Cron scheduling strategy once a day?

How long does it take for this daily listing to complete?

what is the range on number of files that may get listed each day?

-

My initial though here is to take the success output from the ListFile processor and route it to a ReplaceText processor.

The replaceText processor would be used to replace the 0 byte content with just the filename. The filename is stored in a FlowFile attribute on each listed file.

93259-screen-shot-2018-11-14-at-92931-am.png

The "Replacement Value" should include a line return:

93262-screen-shot-2018-11-14-at-94015-am.png

The you would route all these listed FlowFiles to a MergeContent processor that create a single merged file with a list (one filename per line).

93260-screen-shot-2018-11-14-at-93652-am.png

Finally, you can route this single FlowFile to a putEmail processor.

The putEmail processor can be configured to place the Content of the FlowFile as an attachement of directly in message text of email generated.

-

The specific configurations of this suggested flows would need to be adjusted based on your answers to original questions.

-

Thank you,

Matt

-

If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link

Re: file watcher and email notification in NIFI.

Master Guru

@Mitthu Wagh

*** Community Forum Tip: Try to avoid starting a new answer in response to an existing answer. Instead use comments to respond to existing answers. There is no guaranteed order to different answer which can make it hard following a discussion.

-

The ListFile processor will maintain state on what files have been listed by previous run.

*** I strongly recommend you are using at a minimum Apache NIFi 1.8. There have been many important improvements to the ListFile processor and numerous bug fixes.

-

So once a day the listFile processor is running. (perfect)

So as far as MergeContent processor configuration goes, you have a couple options.

1. Schedule it to run daily at some point in time after you daily listFile runs at 9 AM. Just make sure you leave enough time to ensure complete listing completed plus some buffer time.

2. Configure your MergeContent processor with an arbitrary large "Minimum number of Entries" value. This value should well exceed any expected number of files that may be listed at any given day. This will prevent MergeContent from merging the FlowFiles before all files have been placed in incoming queue to MergeContent. The ListFile will output all listed FlowFiles at once, but their will be microseconds/milliseconds difference in time they land in connection to MergeContent. Setting a Max Bin age value will force merging of bin regardless of number of entries. So this setting dictates how long to give for FlowFiles to start and finish getting added to the inbound connection for mergeContent.

-

Thank you,

Matt

Re: file watcher and email notification in NIFI.

New Contributor

Hi Matt,

Are you listing and fetching the new files or just listing them?

>>I want only the list of file names(new and updated)

Are you running this listing using a Cron scheduling strategy once a day?

>>Yes, once in a day (9AM).

what is the range on number of files that may get listed each day?

>>The file range in not fixed (stuck here only)

Can you suggest something, if range on number of files is not fixed then how to list out all file names.

Thanks