Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi: Rolling Window Restful API Polling

Solved Go to solution
Highlighted

NiFi: Rolling Window Restful API Polling

Hello,

Have been recently investing a lot of time in learning and using NiFi for a multitude of cases. On in particular we have a hard time understand how to accomplish is pulling events from a remote restful API endpoint with a rolling time window.

We have successfully used invokehttp processor to pull the desired events using a specific query that includes a time parameter (in this case its an epoch timestamp). While this is great the timestamp is static and the goal would be to update the timestamp with the time in which the last successful invokhttp request occurred.

Does anyone know how this would be accomplished within NiFi?

Thanks

~Regards

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: NiFi: Rolling Window Restful API Polling

Hi @Matthew Gaetano

You should be able to do this using an UpdateAttribute with the state feature. You can add an UpdateAttribute to the flow:

42756-invoke.png

And configure it as follows:

42757-conf.png

This way, each flow file going to InvokeHttp will have two attributes:

  • Timestamp : the current time
  • previousTimestamp : the time of the previous execution. You can use this in your InvokeHttp.

Here's an example of attributes that InvokeHttp will have access to (in my case I execute this every minute as you can see)42758-2.png

Is this what you are looking for?

View solution in original post

3 REPLIES 3
Highlighted

Re: NiFi: Rolling Window Restful API Polling

Hi @Matthew Gaetano

You should be able to do this using an UpdateAttribute with the state feature. You can add an UpdateAttribute to the flow:

42756-invoke.png

And configure it as follows:

42757-conf.png

This way, each flow file going to InvokeHttp will have two attributes:

  • Timestamp : the current time
  • previousTimestamp : the time of the previous execution. You can use this in your InvokeHttp.

Here's an example of attributes that InvokeHttp will have access to (in my case I execute this every minute as you can see)42758-2.png

Is this what you are looking for?

View solution in original post

Highlighted

Re: NiFi: Rolling Window Restful API Polling

Hi @Abdelkrim Hadjidj

For the most part yes, especially since i didn't know how state functioned in XYZ processors. However in this setup the timestamp is always updated regardless of the success or failure of the invokehttp processor. In a failure scenario the poll would potential miss a period of time (and thus any events/data within) so the timestamp can only be updated under a success condition of a request.

At the moment the only thing i can think of is using the distributed cache feature, but that seems like overkill ...

Highlighted

Re: NiFi: Rolling Window Restful API Polling

Hi @Matthew Gaetano

There's another feature in the UpdateProcessor that you can use which Rules : https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.4.0/or...

This feature is available in the advanced settings and you can use to define conditions and actions. So you can say if condition than update timestamps, else do not.

Look at my answer to this question to see how you can use it : https://community.hortonworks.com/questions/140060/nifi-how-to-load-a-value-in-memory-one-time-from-...

Don't have an account?
Coming from Hortonworks? Activate your account here