Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NiFi: Rolling Window Restful API Polling

avatar

Hello,

Have been recently investing a lot of time in learning and using NiFi for a multitude of cases. On in particular we have a hard time understand how to accomplish is pulling events from a remote restful API endpoint with a rolling time window.

We have successfully used invokehttp processor to pull the desired events using a specific query that includes a time parameter (in this case its an epoch timestamp). While this is great the timestamp is static and the goal would be to update the timestamp with the time in which the last successful invokhttp request occurred.

Does anyone know how this would be accomplished within NiFi?

Thanks

~Regards

1 ACCEPTED SOLUTION

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
3 REPLIES 3

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar

Hi @Abdelkrim Hadjidj

For the most part yes, especially since i didn't know how state functioned in XYZ processors. However in this setup the timestamp is always updated regardless of the success or failure of the invokehttp processor. In a failure scenario the poll would potential miss a period of time (and thus any events/data within) so the timestamp can only be updated under a success condition of a request.

At the moment the only thing i can think of is using the distributed cache feature, but that seems like overkill ...

avatar

Hi @Matthew Gaetano

There's another feature in the UpdateProcessor that you can use which Rules : https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.4.0/or...

This feature is available in the advanced settings and you can use to define conditions and actions. So you can say if condition than update timestamps, else do not.

Look at my answer to this question to see how you can use it : https://community.hortonworks.com/questions/140060/nifi-how-to-load-a-value-in-memory-one-time-from-...