Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NiFi: Rolling Window Restful API Polling

avatar

Hello,

Have been recently investing a lot of time in learning and using NiFi for a multitude of cases. On in particular we have a hard time understand how to accomplish is pulling events from a remote restful API endpoint with a rolling time window.

We have successfully used invokehttp processor to pull the desired events using a specific query that includes a time parameter (in this case its an epoch timestamp). While this is great the timestamp is static and the goal would be to update the timestamp with the time in which the last successful invokhttp request occurred.

Does anyone know how this would be accomplished within NiFi?

Thanks

~Regards

1 ACCEPTED SOLUTION

avatar

Hi @Matthew Gaetano

You should be able to do this using an UpdateAttribute with the state feature. You can add an UpdateAttribute to the flow:

42756-invoke.png

And configure it as follows:

42757-conf.png

This way, each flow file going to InvokeHttp will have two attributes:

  • Timestamp : the current time
  • previousTimestamp : the time of the previous execution. You can use this in your InvokeHttp.

Here's an example of attributes that InvokeHttp will have access to (in my case I execute this every minute as you can see)42758-2.png

Is this what you are looking for?

View solution in original post

3 REPLIES 3

avatar

Hi @Matthew Gaetano

You should be able to do this using an UpdateAttribute with the state feature. You can add an UpdateAttribute to the flow:

42756-invoke.png

And configure it as follows:

42757-conf.png

This way, each flow file going to InvokeHttp will have two attributes:

  • Timestamp : the current time
  • previousTimestamp : the time of the previous execution. You can use this in your InvokeHttp.

Here's an example of attributes that InvokeHttp will have access to (in my case I execute this every minute as you can see)42758-2.png

Is this what you are looking for?

avatar

Hi @Abdelkrim Hadjidj

For the most part yes, especially since i didn't know how state functioned in XYZ processors. However in this setup the timestamp is always updated regardless of the success or failure of the invokehttp processor. In a failure scenario the poll would potential miss a period of time (and thus any events/data within) so the timestamp can only be updated under a success condition of a request.

At the moment the only thing i can think of is using the distributed cache feature, but that seems like overkill ...

avatar

Hi @Matthew Gaetano

There's another feature in the UpdateProcessor that you can use which Rules : https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.4.0/or...

This feature is available in the advanced settings and you can use to define conditions and actions. So you can say if condition than update timestamps, else do not.

Look at my answer to this question to see how you can use it : https://community.hortonworks.com/questions/140060/nifi-how-to-load-a-value-in-memory-one-time-from-...