Skip to main content
scheeri
scheeri
Silver ⭐️
June 29, 2023
Solved

Apache Nifi: how to ensure record order between processors

Dear Community!

 

I have a simple flow in Apache NiFi 1.20:

1. QueryDatabaseTableRecord -> 2. SplitAvro -> 3. ConvertAvroToJson -> 4. LogAttribute -> 5. PutDatabaseRecord

 

Flow descrition:

1. Read records from a table order by "updated" column (a datetime with millisecs).

2. put each of records into single flowfile

3. Convert a record into JSON

4. Log JSON into log file.

5. insert a record to different database

Each of processors have only 1 single concurrent thread on SETTING page!

 

Everything works fine, except sometime the order of the records will be changed according to "updated" column.

 

Could you please find me the cause? How can be ensured the order of records right?

 

cjervis
cjervisAnswer
Community Manager
June 29, 2023

Hi @scheeri, I'm not an expert but maybe this well written reply by @MattWho will get you closer. https://community.cloudera.com/t5/Support-Questions/Ensuring-of-order-of-flow-files-in-Nifi/m-p/314305/highlight/true#M225961 

Keep the questions coming
View in thread

3 replies

cjervis
Community Manager
cjervisAnswer
Community Manager
June 29, 2023

Hi @scheeri, I'm not an expert but maybe this well written reply by @MattWho will get you closer. https://community.cloudera.com/t5/Support-Questions/Ensuring-of-order-of-flow-files-in-Nifi/m-p/314305/highlight/true#M225961 

Translate
Keep the questions coming
scheeri
scheeriAuthor
Silver ⭐️
June 29, 2023

Thank you for your answer!

It was not tested by me, but probably there was problem with prioritizing, because it has to be set directly by hand. It is strange, because a queue should has FIFO prioritized by default, i think.

Translate
MattWho
Titanium ⭐️⭐️⭐️
MattWho
Titanium ⭐️⭐️⭐️
June 29, 2023

@scheeri 

 

FIFO is the default priority of a connection queue; however, if you have more then one concurrent task on a processor, multiple FlowFiles from the source connection can be executed about concurrently.  This means that one of those concurrent execution may complete before the other leading to FlowFile no longer being in same order in follow-on connections.

 

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

Translate
Terms & ConditionsAccessibility statement