Skip to main content
kumsath
kumsath
Gold ⭐️
March 28, 2023
Question

SENDING HTTP RESPONSE PART BY PART(PAGINATION) IN NIFI

Hi,

I am receiving and HTTP call in NIFI. Which does the below job:

  • Execute the sql and fetch result
  • Convert the result in to JSON(there are 100 or more number of records)
  • Send back the result as HTTPRESPONSE.

Now the issue is that if i want to sent the data part by part( Using pagination mode) . For an example:

  • Receiving the HTTP call . In the call we will be receiving page number. Say it is 5 pages.
  • So if the total record is 100 then first response will be 20 records(100/5).  
  • First response will have 20 records, second response will have another 20. Like that 5 times
  • Atrans_issue.PNG

How can i handle this in NIFI ? Please provide your thoughts on the same. Your help will be much appreciated

3 replies

cotopaul
cotopaul
Platinum ⭐️
March 28, 2023

hi @kumsath,

If you want to split your output from ExecuteSQL, you could try replacing ConvertAvroToJson with a SplitRecord in which you will configure the Record Reader as AVRO and the Record Writer as JSON, whereas the Records per Split can be set by you to any value. In this way, your ExecuteSQL results are always split into smaller chunks, based on your desired output. Of course, you can use the NiFi Expression Language and calculate the Records per Split dynamically and so on.

Translate
kumsath
kumsathAuthor
Gold ⭐️
March 29, 2023

That is fine. But how to send this part by part output in the HTTP response?

Translate
cotopaul
cotopaul
Platinum ⭐️
March 29, 2023

To be honest I do not really understand your question. If you split your original file into smaller chunks, you will have X flowfiles instead of a single flowfile. Those flowfiles will then go sequentially in your HTTP response Processor, based on your load balance strategy.

Translate
Terms & ConditionsAccessibility statement