InvokeHTTP with pagination, MergeContent JSON to single FF - need some help

robdisseldorp — Wed, 08 Nov 2023 22:07:50 GMT

Hello,

I am just starting to learn NiFi and I really appreciate the power of this great tool.

But I am struggling for several evenings to get what seems a simple flow to work correctly. I have reviewed several samples on how to make an API call and use pagination to retrieve difference pages. This seems to work but I get a unique FF every time. Usign MergeCOntent I was hoping I could create a combined file containing JSON array of all FF content.

It seems so simple but I am missing something. Could anyone point me in the right direction?

Logic:

1) Create FF with starting values limit (500) and start (0)

2) InvokeHTTP with URL and starting values

3) Update filename with filename prefix (object_type) and current date YYYY-MM-DD

4) Check for additional pages in Pagination group (pagination seems to work, getting multiple pages with different JSON data)

5) Iterate through InvokeHTTP with new start offset for next page

6) Using both next_page and Output output streams to ensure all pages are provided

7) MergeContent: try to merge all generate FF (by InvokeHTTP) and group by Correlation Attribute filename (should be the same during each iteration)

😎 UpdateAttribute to set final filename including hh:mm:ss

9) write single FF with merged JSON array

Problem: I still get separate FF after MergeContent.

Seems the configuration of MergeContent do not create single FF.

Re: InvokeHTTP with pagination, MergeContent JSON to single FF - need some help

DianaTorres — Wed, 08 Nov 2023 22:18:06 GMT

@robdisseldorp Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi expert @steven-matison who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

question Re: InvokeHTTP with pagination, MergeContent JSON to single FF - need some help in Support Questions

InvokeHTTP with pagination, MergeContent JSON to single FF - need some help

Re: InvokeHTTP with pagination, MergeContent JSON to single FF - need some help