Monitor Activity Issue

[#1]

Hello Guys,

I am facing some issue with Monitor Activity processor . We are extracting data by using QueryDatabaseTable processor its mean we are extracting only delta records but some time delta is not coming so we have used Monitor Activity processor in this scenario to generate one dummy flow file for keep moving my down stream flow but the problem is our Nifi is running on 3 server in production and we have scheduled QueryDatabaseTable processor at 9 AM UTC and Monitor Activity processor at 10 AM UTC and when the flow started at 9 am QueryDatabaseTable processor runs at different server and it has extracted data and it send to down stream and loaded to hive now at 10 am Monitor Activity processor also got executed and generated a flow file because its has executed on different server is there any way to run both processor on same server or any setting inside Monotor activity processor?

@shu

Regards,

Shantanu.

[#2]

The MonitorActivity processor has a "Monitoring Scope" property that determines whether to look at a node or cluster level for flow activity. It defaults to "node," but should usually be set to "cluster" in a clustered environment to make sure that all instances of the monitored flow are checked for activity.

There's a bit more information here:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.6.0/org.apache...

[#3]

Thank you We are already using Monitoring Scope = cluster but still the problem is there.

[#4]

Can you share the configuration of your MonitorActivity processor?

[#5]

The behavior of MonitorActivity is pretty flexible, so there are some more options. If you have it set up like this (just using GenerateFlowFile in place of QueryDatabaseTable here):

then MonitorActivity will send an alert (inactive) message once GenerateFlowFile has been idle for the Threshold Duration. In this example, GenerateFlowFile is set to trigger every 60 seconds on all nodes, and MonitorActivity is set with a 20-second ThresholdDuration on the Primary Node with cluster scope.

What this will generate is:

0 sec - GenerateFlowFile generates a batch of flowfiles, MonitorActivity flows the batch through to PutHDFS via SUCCESS path (on any node)

20 sec - MonitorAttribute generates inactive alert flowfile via INACTIVE path (on primary node)

60 sec - GenerateFlowFile generates a batch of flowfiles, MonitorActivity flows the batch through to PutHDFS via SUCCESS path (on any node), MonitorActivity generates ACTIVITY.RESTORED flowfile via ACTIVITY.RESTORED path (on primary node)

80 sec - MonitorAttribute generates inactive alert flowfile via INACTIVE path (on primary node)

120 sec - GenerateFlowFile generates a batch of flowfiles, MonitorActivity flows the batch through to PutHDFS via SUCCESS path (on any node), MonitorActivity generates ACTIVITY.RESTORED flowfile via ACTIVITY.RESTORED path (on primary node)

etc.

Is this the behavior you're looking for? If not, what is the specific desired behavior?

[#6]

Were you able to resolve your issue with the MonitorActivity processor @Shantanu kumar?

[#7]

here is the details.

[#8]

It looks like the Threshold Duration is set to 1 minute in the configuration pictured. In that case, the processor started at 10:00 would trigger an inactivity message if the upstream processor had not generated a flowfile for the past 60 seconds. All flowfiles passed to the MonitorActivity processor will also be passed through to the Success relationship when the processor starts.

If instead the desired action is to send a dummy flowfile only if there has been no activity on the QueryDatabaseTable processor for one hour, scheduling both to run at 9:00 with a 60 minute threshold on the MonitorActivity should work.

[#9]

Thanks you @William Brooks I will apply your concept and let you know the result.

Thanks,

Shantanu

[#10]

Hello @William Brooks i tired your options but looks like it not working.It has executed on the same time and 2 flow file inserted in to down stream .