NiFi - ListDatabaseTables processor failure - is there a way to detect failures?

[#1]

Hi,

We are kicking off a NiFi flow with the ListDatabaseTables processor and we experienced an incident where this processor failed to run. This was due to a network adapter error which seems to have been caused by a restart to NiFi, re-enabling the controller service fixed this issue.

However, this is a scheduled job and I would like the flow to be able to respond to an error. The ListDatabaseTables processor will not produce a flowfile when it fails, it has no failure path.

Is there a way in which I could add a cron driven processor to send an email if a flowfile doesn't flow through it by a certain time? I can't think of any other solutions that could be possible within NiFi.

Any ideas would be appreciated.

Thanks,

Peter

[#2 · Accepted solution]

@Peter Murray

You can use the MonitorActivity processor in the connection out of the ListDatabaseTables processor, and set the interval on how long to wait for a flow file, if no flow file is generated in the configured time period, then the processor will generate a lack of activity flow file and that can be connected to a PutEmail processor. When a flow file does arrive, you can use the processor to notify you of that also. The flow file out of the ListDatabaseTables processor will go out the success relationship and can be connected to the next processor in your normal flow. A simple example below:

[#3]

Hi @Wynner ,

I found something similar to this suggestion yesterday, but thank you. I do have a follow question from an issue I noticed this morning.

So I attempted to use a cron schedule to kick off the MonitorActivity which I specified to run 1 minute after the ListDataTables. I thought this was a good way to implement it. Checking the flow this morning, it seems only 50 out of the 800 flowfiles (800 tables from source DB) got through to success while the others just sat in the queue.

I guess this must be some limitation on MonitorActivity's part when scheduled through a cron expression. Would the best way to implement this processor be to set the Threshold Duration to 24hrs (from an hour after the flow should have run) as the flow is daily batch?

[#4]

@Peter Murray

That is one way to configure it.

The batching of 50 flow files is how the MonitorActivity processor works, no matter what Scheduling Strategy is implemented. I would change the Scheduling Strategy to Timer driven, and have it run once a minute, with the Threshold Duration to 24 hours, as you suggested.