Hi All,
I am looking for an easier solution, I have nifi 1.28.1 in my system
I have an invoke http process that which queries an application on an ID , it returns an array of json objects of the same kind in a paginated way. Each response contains an array of 50 json objects which is default, (of couse, the page size can be increased)
There can be more that 100000 json objects retrieved from the application. The lenght of each json object will remain same. it just has the id and its status
I want to generate a summary file which contain all the json objects and save it into an SFTP location.
I am looking for a best approach to combine all the pages and save it into SFTP.
I tried using the Merge Content Processor, but that did not work cause the first page response was not stored anywhere when the second page came in.
Would it be makes sense to store and the array in a distributed cache, merge the array with next page and so forth till all the pages are complete? The distributed cache location can be pointed to a PVC
Or would it be a better solution if i try to store in SFTP, read the same file from SFTP, append new page and overwrite the summary file with the new array, till all the pages are complete?
Looking for some advice here. Thank you
