InvokeHTTP with pagination

[#1]

Hi All,

I'm trying to use InvokeHTTP to query a REST service that requires me to paginate through the results. Pagination is done through two url parameters, limit and offset. The first time through, limit=100, offset=0. If limit number of results are returned, I need to set offset to 100 and then query again. I think I'm close to getting the flow working but as you can see below, the request seems to sit in the queue after "Set Recurring Pagination Parameters" step. Any ideas? Am I missing a step? Are there any example flows out there on how to do this?

EvaluateJsonPath pulls out a field from the resulting json that says how many results are returned. RouteOnAttribute sets more_pages if num_results = limit. Set Recurring Pagination Parameters modifies the variable used in the url of InvokeHTTP.

Thanks in advance!

[#2 · Accepted solution]

@Frank Maritato

You have the good approach, and IMO it should work. I have followed a similar approach in this blog:

https://pierrevillard.com/2016/04/04/analyze-flickr-account-using-apache/

In your screenshot, all your processors are stopped. When started do you have errors in your InvokeHTTP processor? Can you inspect the requests sent to be sure the increment is done correctly?

Hope this helps.

[#3]

Ah Thanks @Pierre Villard! Nice blog post. This is good validation to what I did 🙂

I just discovered that my InvokeHTTP step had a longer schedule defined from earlier when I was testing that by itself. When I changed it to 0 sec it started paginating through the data like I expected. Thanks again!

[#4]

I call the OpenAQ rest service and increment the pages

https://github.com/tspannhw/flank-airquality/blob/main/flows/AirQuality.json

[#5]

Also I had one with a Kudu cache for calling Daily Med

https://github.com/tspannhw/ApacheConAtHome2020/tree/main/flows/DailyMed

https://www.datainmotion.dev/2021/01/flank-using-apache-kudu-as-cache-for.html