I am working in an AWS environment with NifI on the core of my ETL process, this is a new installation so I have no monitoring setup yet.
Because this is a fresh installation I want to hear the practices you use to monitor NiFi, I used to put some Slack messages for FAIL and SUCCESS but there are very long flows and I don't seem viable putting a slack call on every possible point of failure, those can be a LOT of processors.
How do you do? insert to mysql to monitor start/end and alarm if something doesn't end? LOT of slack/mail processors? Dashboards (how do you feed them)? any suggestions?
You might want to look at using the Site-To-Site Bulletin Reporting Task. Anytime a component produces an ERROR or WARN log output (For example when a FlowFile is routed to a processor's failure relationship), a bulletin is produced on the component. The "SiteToSiteBulletinReportingTask" would get that bulletin and send it via the S2S protocol to another NiFi instance/cluster where you would build a notification dataflow using for example the putEmail processor. Keep in mind that having a different NiFi is not required, you can have the "SiteToSiteBulletinReportingTask" just sent the bulletins to a flow on the same NiFi that your are monitoring. This scales well as new components are added to your canvas and does not require separate monitoring on every flow.