SENDING HTTP RESPONSE PART BY PART(PAGINATION) IN NIFI

[#1]

Hi,

I am receiving and HTTP call in NIFI. Which does the below job:

• Execute the sql and fetch result
• Convert the result in to JSON(there are 100 or more number of records)
• Send back the result as HTTPRESPONSE.

Now the issue is that if i want to sent the data part by part( Using pagination mode) . For an example:

• Receiving the HTTP call . In the call we will be receiving page number. Say it is 5 pages.
• So if the total record is 100 then first response will be 20 records(100/5).  
• First response will have 20 records, second response will have another 20. Like that 5 times
• 

How can i handle this in NIFI ? Please provide your thoughts on the same. Your help will be much appreciated

[#2]

hi @kumsath,

If you want to split your output from ExecuteSQL, you could try replacing ConvertAvroToJson with a SplitRecord in which you will configure the Record Reader as AVRO and the Record Writer as JSON, whereas the Records per Split can be set by you to any value. In this way, your ExecuteSQL results are always split into smaller chunks, based on your desired output. Of course, you can use the NiFi Expression Language and calculate the Records per Split dynamically and so on.

[#3]

That is fine. But how to send this part by part output in the HTTP response?

[#4]

To be honest I do not really understand your question. If you split your original file into smaller chunks, you will have X flowfiles instead of a single flowfile. Those flowfiles will then go sequentially in your HTTP response Processor, based on your load balance strategy.