Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Support Questions

Find answers, ask questions, and share your expertise
Announcements
Share your experience with Cloudera on G2 and get a $25 Amazon Gift card.
Hi, I'm CLEO! Something exciting is coming to the Community. Stay Tuned!

Sending an email whenever the Queue is Full- Nifi

avatar
Explorer

Greetings, 

 I wanted to know how to set up a process where one will receive an email when ever a queue reaches its threshold limit , in apache nifi.

Thanks in advance.

 

 

 

 

1 ACCEPTED SOLUTION

avatar
Super Guru

@P_Rat98   Per our PM discussion,  in your flow use DetectDuplicate before sending an email.   This should rate limit the # of messages you send based on your configuration of detectDuplicate.  Additionally when this is linked in your flow, and duplicates are auto-terminated it will drain the flow and stop it from filling up the queue.    Also as suggested you can chose to retain the duplicates, but move them into a much bigger Queue which isnt going to back up the main flow.  Then once you see the email, you can go look at flow, see what flowfiles were causing issues, and take some corrective action.

 

If you really need to monitor flow for a queue being full,  you would need to use the nifi API to check the status of the queue.  This maybe more work than it is worth, when you can solve as above much easier.  However, i would recommend you check the api out,  there are a lot of api capabilities and I am beginning to use nifi api calls within my flow to monitor, stop, start, and take actions automatically that would normally require a human doing them in the UI.

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.

 

Thanks,

Steven

View solution in original post

1 REPLY 1

avatar
Super Guru

@P_Rat98   Per our PM discussion,  in your flow use DetectDuplicate before sending an email.   This should rate limit the # of messages you send based on your configuration of detectDuplicate.  Additionally when this is linked in your flow, and duplicates are auto-terminated it will drain the flow and stop it from filling up the queue.    Also as suggested you can chose to retain the duplicates, but move them into a much bigger Queue which isnt going to back up the main flow.  Then once you see the email, you can go look at flow, see what flowfiles were causing issues, and take some corrective action.

 

If you really need to monitor flow for a queue being full,  you would need to use the nifi API to check the status of the queue.  This maybe more work than it is worth, when you can solve as above much easier.  However, i would recommend you check the api out,  there are a lot of api capabilities and I am beginning to use nifi api calls within my flow to monitor, stop, start, and take actions automatically that would normally require a human doing them in the UI.

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.

 

Thanks,

Steven