Handling multiple email accounts in NiFi using ConsumePOP3 (equivalent to looping in Java)

[#1]

I am working with Apache NiFi and trying to replicate a pattern that we had done in existing system.

In existing system, we have multiple (around 10) email accounts, and we iterate over them using a loop, connecting to each mailbox, reading emails, and processing them.

In NiFi, I have configured a ConsumePOP3 processor and it works fine for a single email account. However, I need to handle multiple email accounts in a scalable and maintainable way. We are not able to connect any processor or something before ConsumePOP3.

[#2]

Hi AlokKumar! 

While I'm not a technical expert on the product, I noticed you're looking to scale your flow for multiple accounts. While you wait for a specialist to reply, you might find some useful starting points in this NiFi Design Pattern article or this discussion on handling multiple requests. 

Hope this helps point you in the right direction!

[#3]

@AlokKumar  

You are correct that the ConsumePOP3 processor does not support an inbound connection to it.    Even if it did, the username and password fields do not support NiFi Expression language allowing you to pass either of those values in from the source FlowFile.   There aren't any other native processors that can support this dynamic credentials use case.   You would need to create a custom script that could be called by scripting processors or create your own custom processor. 

• ExecuteScript
• ExecuteProcess
• ExecuteGroovyScript

The reason processors like ConsumePOP3 do not support inbound connections is because they are designed to execute continuously on a run schedule and produce an individual FlowFile for each new email message consumed.   So supporting an inbound connection raises the question... What do you do with the source FlowFile that you would use as the trigger?  Then you also have the challenge of continuously consumption.  You would need to keep producing an input FlowFile for each email account to make sure you keep consuming from each source account.   Plus this processor does not write any attributes to outbound FlowFile to distinguish which account message came from. 

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt