question Re: Does NiFi preserves order of transactions in a Cluster in Support Questions

Does NiFi preserves order of transactions in a Cluster

mayank_rathi — Wed, 05 Jul 2017 02:37:50 GMT

Hello All,

Does NiFi preserve order of transactions in a cluster? Is their a way to guarantee that Inserts will be processed before Updates? I can use Kafka Connect with single partition and 3 replicas to ensure HA and Transaction Order. Wondering if their is anyway to achieve this in a NiFi cluster.

Thanks!!

Re: Does NiFi preserves order of transactions in a Cluster

ssahi — Wed, 05 Jul 2017 04:11:15 GMT

Hi @M R

NiFi can do this. You may want to look at the EnforceOrder processor in the latest NiFi version. In a clustered enviornment, you can have most of your processors distributed throughout the cluster but one of the last processors in the flow could be EnforceOrder, which should be run only on a Primary Node to ensure that the order is maintained. Also, use FIFO prioritization with it.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.3.0/org.apache.nifi.processors.standard.EnforceOrder/index.html

Re: Does NiFi preserves order of transactions in a Cluster

MattWho — Thu, 06 Jul 2017 01:33:36 GMT

@M R

Using a single partition in Kafka is similar to using a single Node in NiFi to maintain order. The enforceOrder processor is a great new addition for enforcing order of FlowFiles, but it will only enforce orders on those FlowFiles that reside on the same node in a NiFi cluster. So if you are trying to enforce processing order of FlowFiles across numerous nodes, this processor will not do that. You would need to get all FlowFiles for which you want to enforce order on to the same node before using this processor.

I don't fully understand you entire use case, but a couple other processor you may want to look at include wait ad notify processors. These are also new in the latest HDF 3.0 and NiFi 1.2.0 releases.

Thanks,

Matt