Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

page through results - InvokeHTML process through NIFI

Highlighted

page through results - InvokeHTML process through NIFI

New Contributor

Hi, I have created a flow using the Invokehtml and by default, the endpoints mentioned in the invoke function will return 300 activity items at a time.  It also has moreResult attribute is if it's true, this means that more results are available. How do I continue to call the endpoint until the moreResult attribute returns false, which means that there are no more results available. ?

4 REPLIES 4
Highlighted

Re: page through results - InvokeHTML process through NIFI

Master Collaborator

@ajay_mamgain200  After the first invokeHtml you should do a check for the moreResults attribute to exist.  If it exists, route that FlowFile back to invokeHttml to run again.  If does not exist or false, go to next step in your flow.

 

Hope this helps..

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven @ DFHZ

Highlighted

Re: page through results - InvokeHTML process through NIFI

New Contributor

Thank you for the quick response @stevenmatison  I have done the exact same however I'm not able to get the new data it's just throwing the same data again and again. image.pngimage.pngimage.png

Re: page through results - InvokeHTML process through NIFI

Master Collaborator

@ajay_mamgain200   Try to go back to the source system, and see if there is something you need to pass on the additional nth calls to indicate you want the "more results" not the initial results.  Sometimes this is another flag, or a range to identify a start/stop point.    If the additional call is quite a bit different, do not loop back to the original invokeHttp, use a separate invokeHttp and loop to this one until done; unless you can make a single InvokeHttp dynamically handle first and additional nth calls.   

 

I would test these calls in postman before attempting in NiFi, so that you know for sure what the first call is, and what 2nd, 3rd, nth is.

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven @ DFHZ

Highlighted

Re: page through results - InvokeHTML process through NIFI

New Contributor

Thank you for the response @stevenmatison I can confirm that we need to make the same API request which I have used in the invokehtml initially. I have done the testing in PowerBI because I can append the data in the list and i was able to get the data. Do you know if we can generate a list nifi and append the data ? It will be difficult to do that in Postman but it does work in Python as well.

 

GeneratedList =
List.Generate(
()=>[i=0, res = FnProcessOnePage(accessTokenParamStr, pagingTokenParamStr)],
each [i]<iterationsNum and [res][Data]<>null,
each [i=[i]+1, res = FnProcessOnePage([res][Access],[res][Next])],
each [res][Data])

Don't have an account?
Coming from Hortonworks? Activate your account here