Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

InvokeHTTP with pagination, MergeContent JSON to single FF - need some help

avatar
New Contributor

Hello,

I am just starting to learn NiFi and I really appreciate the power of this great tool.

But I am struggling for several evenings to get what seems a simple flow to work correctly. I have reviewed several samples on how to make an API call and use pagination to retrieve difference pages. This seems to work but I get a unique FF every time. Usign MergeCOntent I was hoping I could create a combined file containing JSON array of all FF content.

It seems so simple but I am missing something. Could anyone point me in the right direction?

Logic:

1) Create FF with starting values limit (500) and start (0)

2) InvokeHTTP with URL and starting values

3) Update filename with filename prefix (object_type) and current date YYYY-MM-DD

4) Check for additional pages in Pagination group (pagination seems to work, getting multiple pages with different JSON data)

5) Iterate through InvokeHTTP with new start offset for next page

6) Using both next_page and Output output streams to ensure all pages are provided

7) MergeContent: try to merge all generate FF (by InvokeHTTP) and group by Correlation Attribute filename (should be the same during each iteration)

😎 UpdateAttribute to set final filename including hh:mm:ss

9) write single FF with merged JSON array

Problem: I still get separate FF after MergeContent.

Seems the configuration of MergeContent do not create single FF.

W

 

Screenshot 2023-11-08 at 22.19.28.pngScreenshot 2023-11-08 at 22.35.06.pngScreenshot 2023-11-08 at 22.35.24.pngScreenshot 2023-11-08 at 22.35.42.pngScreenshot 2023-11-08 at 22.35.55.pngScreenshot 2023-11-08 at 22.36.01.pngScreenshot 2023-11-08 at 22.36.09.png

 

 

1 REPLY 1

avatar
Community Manager

@robdisseldorp Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi expert @steven-matison  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.


Regards,

Diana Torres,
Community Moderator


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community: