Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How do you monitor NiFi flows?

How do you monitor NiFi flows?

Contributor

I am working in an AWS environment with NifI on the core of my ETL process, this is a new installation so I have no monitoring setup yet.

Because this is a fresh installation I want to hear the practices you use to monitor NiFi, I used to put some Slack messages for FAIL and SUCCESS but there are very long flows and I don't seem viable putting a slack call on every possible point of failure, those can be a LOT of processors.

 

How do you do? insert to mysql to monitor start/end and alarm if something doesn't end? LOT of slack/mail processors? Dashboards (how do you feed them)? any suggestions?

 

Thank you!

1 REPLY 1
Highlighted

Re: How do you monitor NiFi flows?

Master Guru

@Alexandros 

 

You might want to look at using the Site-To-Site Bulletin Reporting Task.  Anytime a component produces an ERROR or WARN log output (For example when a FlowFile is routed to a processor's failure relationship), a bulletin is produced on the component.  The "SiteToSiteBulletinReportingTask" would get that bulletin and send it via the S2S protocol to another NiFi instance/cluster where you would build a notification dataflow using for example the putEmail processor.  Keep in mind that having a different NiFi is not required, you can have the "SiteToSiteBulletinReportingTask" just sent the bulletins to a flow on the same NiFi that your are monitoring.   This scales well as new components are added to your canvas and does not require separate monitoring on every flow.

 

Hope this helps,

Matt

Don't have an account?
Coming from Hortonworks? Activate your account here