How do you monitor NiFi flows?

[#1]

I am working in an AWS environment with NifI on the core of my ETL process, this is a new installation so I have no monitoring setup yet.

Because this is a fresh installation I want to hear the practices you use to monitor NiFi, I used to put some Slack messages for FAIL and SUCCESS but there are very long flows and I don't seem viable putting a slack call on every possible point of failure, those can be a LOT of processors.

How do you do? insert to mysql to monitor start/end and alarm if something doesn't end? LOT of slack/mail processors? Dashboards (how do you feed them)? any suggestions?

Thank you!

[#2]

@Alexandros 

You might want to look at using the Site-To-Site Bulletin Reporting Task.  Anytime a component produces an ERROR or WARN log output (For example when a FlowFile is routed to a processor's failure relationship), a bulletin is produced on the component.  The "SiteToSiteBulletinReportingTask" would get that bulletin and send it via the S2S protocol to another NiFi instance/cluster where you would build a notification dataflow using for example the putEmail processor.  Keep in mind that having a different NiFi is not required, you can have the "SiteToSiteBulletinReportingTask" just sent the bulletins to a flow on the same NiFi that your are monitoring.   This scales well as new components are added to your canvas and does not require separate monitoring on every flow.

Hope this helps,

Matt