question Re: SENDING HTTP RESPONSE PART BY PART(PAGINATION) IN NIFI in Support Questions

SENDING HTTP RESPONSE PART BY PART(PAGINATION) IN NIFI

kumsath — Tue, 28 Mar 2023 13:57:42 GMT

Hi,

I am receiving and HTTP call in NIFI. Which does the below job:

Execute the sql and fetch result
Convert the result in to JSON(there are 100 or more number of records)
Send back the result as HTTPRESPONSE.

Now the issue is that if i want to sent the data part by part( Using pagination mode) . For an example:

Receiving the HTTP call . In the call we will be receiving page number. Say it is 5 pages.
So if the total record is 100 then first response will be 20 records(100/5).
First response will have 20 records, second response will have another 20. Like that 5 times

How can i handle this in NIFI ? Please provide your thoughts on the same. Your help will be much appreciated

Re: SENDING HTTP RESPONSE PART BY PART(PAGINATION) IN NIFI

cotopaul — Tue, 28 Mar 2023 14:52:21 GMT

hi @kumsath,

If you want to split your output from ExecuteSQL, you could try replacing ConvertAvroToJson with a SplitRecord in which you will configure the Record Reader as AVRO and the Record Writer as JSON, whereas the Records per Split can be set by you to any value. In this way, your ExecuteSQL results are always split into smaller chunks, based on your desired output. Of course, you can use the NiFi Expression Language and calculate the Records per Split dynamically and so on.

Re: SENDING HTTP RESPONSE PART BY PART(PAGINATION) IN NIFI

kumsath — Wed, 29 Mar 2023 05:30:08 GMT

That is fine. But how to send this part by part output in the HTTP response?

Re: SENDING HTTP RESPONSE PART BY PART(PAGINATION) IN NIFI

cotopaul — Wed, 29 Mar 2023 13:07:07 GMT

To be honest I do not really understand your question. If you split your original file into smaller chunks, you will have X flowfiles instead of a single flowfile. Those flowfiles will then go sequentially in your HTTP response Processor, based on your load balance strategy.