NiFi: Rolling Window Restful API Polling

[#1]

Hello,

Have been recently investing a lot of time in learning and using NiFi for a multitude of cases. On in particular we have a hard time understand how to accomplish is pulling events from a remote restful API endpoint with a rolling time window.

We have successfully used invokehttp processor to pull the desired events using a specific query that includes a time parameter (in this case its an epoch timestamp). While this is great the timestamp is static and the goal would be to update the timestamp with the time in which the last successful invokhttp request occurred.

Does anyone know how this would be accomplished within NiFi?

Thanks

~Regards

[#2 · Accepted solution]

Hi @Matthew Gaetano

You should be able to do this using an UpdateAttribute with the state feature. You can add an UpdateAttribute to the flow:

And configure it as follows:

This way, each flow file going to InvokeHttp will have two attributes:

• Timestamp : the current time
• previousTimestamp : the time of the previous execution. You can use this in your InvokeHttp.

Here's an example of attributes that InvokeHttp will have access to (in my case I execute this every minute as you can see)

Is this what you are looking for?

[#3]

Hi @Abdelkrim Hadjidj

For the most part yes, especially since i didn't know how state functioned in XYZ processors. However in this setup the timestamp is always updated regardless of the success or failure of the invokehttp processor. In a failure scenario the poll would potential miss a period of time (and thus any events/data within) so the timestamp can only be updated under a success condition of a request.

At the moment the only thing i can think of is using the distributed cache feature, but that seems like overkill ...

[#4]

Hi @Matthew Gaetano

There's another feature in the UpdateProcessor that you can use which Rules : https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.4.0/or...

This feature is available in the advanced settings and you can use to define conditions and actions. So you can say if condition than update timestamps, else do not.

Look at my answer to this question to see how you can use it : https://community.hortonworks.com/questions/140060/nifi-how-to-load-a-value-in-memory-one-time-from-...