Apache Nifi: how to ensure record order between processors

[#1]

Dear Community!

I have a simple flow in Apache NiFi 1.20:

1. QueryDatabaseTableRecord -> 2. SplitAvro -> 3. ConvertAvroToJson -> 4. LogAttribute -> 5. PutDatabaseRecord

Flow descrition:

1. Read records from a table order by "updated" column (a datetime with millisecs).

2. put each of records into single flowfile

3. Convert a record into JSON

4. Log JSON into log file.

5. insert a record to different database

Each of processors have only 1 single concurrent thread on SETTING page!

Everything works fine, except sometime the order of the records will be changed according to "updated" column.

Could you please find me the cause? How can be ensured the order of records right?

[#2 · Accepted solution]

Hi @scheeri, I'm not an expert but maybe this well written reply by @MattWho will get you closer. https://community.cloudera.com/t5/Support-Questions/Ensuring-of-order-of-flow-files-in-Nifi/m-p/3143... 

[#3]

Thank you for your answer!

It was not tested by me, but probably there was problem with prioritizing, because it has to be set directly by hand. It is strange, because a queue should has FIFO prioritized by default, i think.

[#4]

@scheeri 

FIFO is the default priority of a connection queue; however, if you have more then one concurrent task on a processor, multiple FlowFiles from the source connection can be executed about concurrently.  This means that one of those concurrent execution may complete before the other leading to FlowFile no longer being in same order in follow-on connections.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt