Hello!
So I am trying to figure out how to work with the EnforceOrder processor.
I have 200 flowfiles and every flowfile has attribute partition_number. Many flowfiles has the same filename.
I wanna use EnforceOrder to merge them accroding to filename and partition_number attribute
but non of the flowfiles ever go to success relationship queue.
Here's my settings:
What am I doing wrong?
@Brenigan
Are you running your dataflow on a standalone NiFi install or a NiFi cluster install?
If a multi node NiFi cluster, are all 200 FlowFiles on the same NiFi node?
Does your partition_number start at 0?
Do you see your FlowFiles getting routes to the overtook relationship after 10 minutes?
Assuming all the following:
1. All FlowFiles are on same NiFi node
2. partition_number starts at "0" and "increments consistently by "1"
3. All FlowFiles have same filename
4. Wait relationship is route via a connection back to the EnforceOrder processor.
You should be seeing:
1. All FlowFiles routed to the "wait" relationship until a FlowFile with attribute "partition_number" equal to "0" is processed which will result in that FlowFile routing to success.
2. Other FlowFiles meeting above 4 criteria will continue to loop through wait until "partition_number" attribute with value "1" is seen and routed to success.
3. If a FlowFile in incremental order is missing, all FlowFiles with a partition_number higher than the next expected integer will continue to route to wait relationship.
4. after the configured "wait timeout" any FlowFile that has been waiting this long will be routed to the "overtook" relationship.
You can right click on a connection holding the FlowFiles and list the queue. From there you can select the "view details" icon to the far left to examine the FlowFiles current attributes. You should see a new attribute "EnforceOrder.expectedOrder" that contains the next expected integer value that the group this FlowFile belongs to is waiting for. You will also find your "partition_number" which will have the current integer for this FlowFile.
If you have your FlowFiles distributed across multiple nodes in a NiFi cluster, you will need to get all FlowFiles with the same "group identifier" moved to the same NiFi node in order to enforce order (you can not enforce order across different nodes in a NiFi cluster). You can accomplish this by editing the connection feeding your enforceOrder processor and under settings select a "Load Balancing Strategy" of "Partition by Attribute" using the "filename" attribute that you are using as your group identifier in the Enforce Order processor.
If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.
Thank you,
Matt
@Brenigan
Are you running your dataflow on a standalone NiFi install or a NiFi cluster install?
If a multi node NiFi cluster, are all 200 FlowFiles on the same NiFi node?
Does your partition_number start at 0?
Do you see your FlowFiles getting routes to the overtook relationship after 10 minutes?
Assuming all the following:
1. All FlowFiles are on same NiFi node
2. partition_number starts at "0" and "increments consistently by "1"
3. All FlowFiles have same filename
4. Wait relationship is route via a connection back to the EnforceOrder processor.
You should be seeing:
1. All FlowFiles routed to the "wait" relationship until a FlowFile with attribute "partition_number" equal to "0" is processed which will result in that FlowFile routing to success.
2. Other FlowFiles meeting above 4 criteria will continue to loop through wait until "partition_number" attribute with value "1" is seen and routed to success.
3. If a FlowFile in incremental order is missing, all FlowFiles with a partition_number higher than the next expected integer will continue to route to wait relationship.
4. after the configured "wait timeout" any FlowFile that has been waiting this long will be routed to the "overtook" relationship.
You can right click on a connection holding the FlowFiles and list the queue. From there you can select the "view details" icon to the far left to examine the FlowFiles current attributes. You should see a new attribute "EnforceOrder.expectedOrder" that contains the next expected integer value that the group this FlowFile belongs to is waiting for. You will also find your "partition_number" which will have the current integer for this FlowFile.
If you have your FlowFiles distributed across multiple nodes in a NiFi cluster, you will need to get all FlowFiles with the same "group identifier" moved to the same NiFi node in order to enforce order (you can not enforce order across different nodes in a NiFi cluster). You can accomplish this by editing the connection feeding your enforceOrder processor and under settings select a "Load Balancing Strategy" of "Partition by Attribute" using the "filename" attribute that you are using as your group identifier in the Enforce Order processor.
If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.
Thank you,
Matt
