Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

InvokeHTTP with pagination

Solved Go to solution

InvokeHTTP with pagination

Contributor

Hi All,

I'm trying to use InvokeHTTP to query a REST service that requires me to paginate through the results. Pagination is done through two url parameters, limit and offset. The first time through, limit=100, offset=0. If limit number of results are returned, I need to set offset to 100 and then query again. I think I'm close to getting the flow working but as you can see below, the request seems to sit in the queue after "Set Recurring Pagination Parameters" step. Any ideas? Am I missing a step? Are there any example flows out there on how to do this?

EvaluateJsonPath pulls out a field from the resulting json that says how many results are returned. RouteOnAttribute sets more_pages if num_results = limit. Set Recurring Pagination Parameters modifies the variable used in the url of InvokeHTTP.

Thanks in advance!

7867-screen-shot-2016-09-22-at-65833-am.png

1 ACCEPTED SOLUTION

Accepted Solutions

Re: InvokeHTTP with pagination

@Frank Maritato

You have the good approach, and IMO it should work. I have followed a similar approach in this blog:

https://pierrevillard.com/2016/04/04/analyze-flickr-account-using-apache/

In your screenshot, all your processors are stopped. When started do you have errors in your InvokeHTTP processor? Can you inspect the requests sent to be sure the increment is done correctly?

Hope this helps.

2 REPLIES 2

Re: InvokeHTTP with pagination

@Frank Maritato

You have the good approach, and IMO it should work. I have followed a similar approach in this blog:

https://pierrevillard.com/2016/04/04/analyze-flickr-account-using-apache/

In your screenshot, all your processors are stopped. When started do you have errors in your InvokeHTTP processor? Can you inspect the requests sent to be sure the increment is done correctly?

Hope this helps.

Highlighted

Re: InvokeHTTP with pagination

Contributor

Ah Thanks @Pierre Villard! Nice blog post. This is good validation to what I did :)

I just discovered that my InvokeHTTP step had a longer schedule defined from earlier when I was testing that by itself. When I changed it to 0 sec it started paginating through the data like I expected. Thanks again!