Created 05-31-2018 02:50 PM
I am facing some issue with Monitor Activity processor . We are extracting data by using QueryDatabaseTable processor its mean we are extracting only delta records but some time delta is not coming so we have used Monitor Activity processor in this scenario to generate one dummy flow file for keep moving my down stream flow but the problem is our Nifi is running on 3 server in production and we have scheduled QueryDatabaseTable processor at 9 AM UTC and Monitor Activity processor at 10 AM UTC and when the flow started at 9 am QueryDatabaseTable processor runs at different server and it has extracted data and it send to down stream and loaded to hive now at 10 am Monitor Activity processor also got executed and generated a flow file because its has executed on different server is there any way to run both processor on same server or any setting inside Monotor activity processor?
Created 05-31-2018 08:50 PM
The MonitorActivity processor has a "Monitoring Scope" property that determines whether to look at a node or cluster level for flow activity. It defaults to "node," but should usually be set to "cluster" in a clustered environment to make sure that all instances of the monitored flow are checked for activity.
There's a bit more information here:
Created 05-31-2018 09:30 PM
Thank you We are already using Monitoring Scope = cluster but still the problem is there.
Created 06-01-2018 01:13 PM
Can you share the configuration of your MonitorActivity processor?
Created on 06-07-2018 02:39 PM - edited 08-17-2019 09:37 PM
The behavior of MonitorActivity is pretty flexible, so there are some more options. If you have it set up like this (just using GenerateFlowFile in place of QueryDatabaseTable here):
then MonitorActivity will send an alert (inactive) message once GenerateFlowFile has been idle for the Threshold Duration. In this example, GenerateFlowFile is set to trigger every 60 seconds on all nodes, and MonitorActivity is set with a 20-second ThresholdDuration on the Primary Node with cluster scope.
What this will generate is:
0 sec - GenerateFlowFile generates a batch of flowfiles, MonitorActivity flows the batch through to PutHDFS via SUCCESS path (on any node)
20 sec - MonitorAttribute generates inactive alert flowfile via INACTIVE path (on primary node)
60 sec - GenerateFlowFile generates a batch of flowfiles, MonitorActivity flows the batch through to PutHDFS via SUCCESS path (on any node), MonitorActivity generates ACTIVITY.RESTORED flowfile via ACTIVITY.RESTORED path (on primary node)
80 sec - MonitorAttribute generates inactive alert flowfile via INACTIVE path (on primary node)
120 sec - GenerateFlowFile generates a batch of flowfiles, MonitorActivity flows the batch through to PutHDFS via SUCCESS path (on any node), MonitorActivity generates ACTIVITY.RESTORED flowfile via ACTIVITY.RESTORED path (on primary node)
Is this the behavior you're looking for? If not, what is the specific desired behavior?
Created 07-12-2018 04:33 PM
Created on 06-01-2018 05:50 PM - edited 08-17-2019 09:37 PM
here is the details.
Created 06-02-2018 12:38 AM
It looks like the Threshold Duration is set to 1 minute in the configuration pictured. In that case, the processor started at 10:00 would trigger an inactivity message if the upstream processor had not generated a flowfile for the past 60 seconds. All flowfiles passed to the MonitorActivity processor will also be passed through to the Success relationship when the processor starts.
If instead the desired action is to send a dummy flowfile only if there has been no activity on the QueryDatabaseTable processor for one hour, scheduling both to run at 9:00 with a 60 minute threshold on the MonitorActivity should work.
Created 06-02-2018 08:50 PM
Created 06-03-2018 07:22 PM
Hello @William Brooks i tired your options but looks like it not working.It has executed on the same time and 2 flow file inserted in to down stream .