question InvokeHTTP with pagination in Archives of Support Questions (Read Only)

InvokeHTTP with pagination

frankmarit — Sun, 18 Aug 2019 11:18:04 GMT

Hi All,

I'm trying to use InvokeHTTP to query a REST service that requires me to paginate through the results. Pagination is done through two url parameters, limit and offset. The first time through, limit=100, offset=0. If limit number of results are returned, I need to set offset to 100 and then query again. I think I'm close to getting the flow working but as you can see below, the request seems to sit in the queue after "Set Recurring Pagination Parameters" step. Any ideas? Am I missing a step? Are there any example flows out there on how to do this?

EvaluateJsonPath pulls out a field from the resulting json that says how many results are returned. RouteOnAttribute sets more_pages if num_results = limit. Set Recurring Pagination Parameters modifies the variable used in the url of InvokeHTTP.

Thanks in advance!

Re: InvokeHTTP with pagination

pvillard — Thu, 22 Sep 2016 21:23:08 GMT

@Frank Maritato

You have the good approach, and IMO it should work. I have followed a similar approach in this blog:

https://pierrevillard.com/2016/04/04/analyze-flickr-account-using-apache/

In your screenshot, all your processors are stopped. When started do you have errors in your InvokeHTTP processor? Can you inspect the requests sent to be sure the increment is done correctly?

Hope this helps.

Re: InvokeHTTP with pagination

frankmarit — Thu, 22 Sep 2016 21:42:09 GMT

Ah Thanks @Pierre Villard! Nice blog post. This is good validation to what I did 🙂

I just discovered that my InvokeHTTP step had a longer schedule defined from earlier when I was testing that by itself. When I changed it to 0 sec it started paginating through the data like I expected. Thanks again!

Re: InvokeHTTP with pagination

TimothySpann — Thu, 27 Apr 2023 16:53:08 GMT

I call the OpenAQ rest service and increment the pages

https://github.com/tspannhw/flank-airquality/blob/main/flows/AirQuality.json

Re: InvokeHTTP with pagination

TimothySpann — Thu, 27 Apr 2023 17:03:02 GMT

Also I had one with a Kudu cache for calling Daily Med

https://github.com/tspannhw/ApacheConAtHome2020/tree/main/flows/DailyMed

https://www.datainmotion.dev/2021/01/flank-using-apache-kudu-as-cache-for.html