@edim2525  Apache NiFi is designed to move data fast and efficiently.  Each processor consumes FlowFiles from an incoming connection with no guaranteed priority/order, but depending on the processors used and concurrency setting, that does not guarantee order of FlowFiles execution.  Nor can order be guaranteed when FlowFiles are spread across nodes in a multi-node NiFi cluster. Now you did not mention if you are a single standalone instance of NiFi or a NiFi cluster.  You also do not mention if you have other NiFi component processors between your consumeKafka and PublishKafka processors. The ConsumeKafka processor consumes messages from a Topic and produces one FlowFile for each message.  For most efficient processing the ConsumeKafka processor would typically be have its concurrent tasks configured to equal or be less than number of partitions on the topic (never more as that causes constant rebalance).  This creates a consumer group allowing each concurrent thread execution to pull data from specific partitions assigned to the consumer group. As an example:  If you had a 3 partition topic and a 3 node NiFi cluster, you would configure your ConsumeKafka with 1 concurrent task (this is a total of 3 concurrent tasks across yoru 3 NiFi nodes). Each NiFi node would would be assigned a different topic partition to consume from.  Now each node in a NiFi cluster executes its own copy of the flow with no knowledge of or regards to what FlowFiles are be processed on other nodes.  This is why guaranteed order in a NiFi cluster is very difficult.  There are options to run some dataflows on Primary node only keeping that dataflow running on only one node.  But even then you can have the primary node election change and NiFi has no way to force or guarantee a specific nodes is assigned or retains the primary node role. Somethings you may try that might work for you... Suggestion 1: 1. Standalone NiFi instance (removes issues that a cluster can introduce when order is important) 2. When using ConsumeKafka, try using EnforceOrder processor after ConsumeKafka configured to order FlowFiles by the offset.  Then make sure that all connections downstream of EnforceOrder processor to PublishKafka use "FirstInFirstOutPrioritizer" (by default no prioritizer is set on a connection). 3. Make sure all processors in this dataflow are configured with only 1 concurrent task to avoid any concurrency that could lead to some FlowFile being processed faster then other impacting queued order in downstream connections. Suggestion 2: 1. Standalone NiFi instance for same reason as above. 2. Use ConsumeKafkaRecord instead of ConsumeKafka.  This produce fewer FlowFiles with each FlowFile containing multiple messages. 3. Use PublishKafkaRecord instead of PublishKafka  4. Make sure that all connections downstream of PublsihKafkaRecord processor to PublishKafkaRecord processor use "FirstInFirstOutPrioritizer" (by default no prioritizer is set on a connection). 5. still use one concurrent task on all processors to avoid concurrent record production. --- This option leads to fewer FlowFile (meaning less overhead and lower chance of messages becoming out of order) and likely faster throughput. Suggestion 3: If your dataflow consists of nothing more than consume and publish and you aer doing nothing with the data between these processor, as @steven-matison mentioned, NiFi may not be the best choice here.  NiFi is designed around faster and efficient movement of data using multi-node clusters  and concurrent thread execution.  Additionally NiFi needs to consume all the messages in to FlowFiles (writing content to content_repository and metatdata to the flowfile_repository) and  records data provenance which adds to overhead. So a purpose built tool for a simple replication task may be better for this use case. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@ChuckE  You can right click on the UpdateAttribute processor and "view state" which will allow you to clear the stored state for that processor without needing to create a new processor. Every dynamic property you add becomes a different local state variable that is added to the FlowFiles as a FlowFile Attribute. Thanks, Matt ... View more

@Fayza  Apache CouchDB looks very limited in its query capabilities. But it does appear that a user in the community created nifi-couchDB components: https://github.com/wasperen/nifi-couchdb You could try adding this nar to your Apache NiFi installation to see if these components can satisfy your use case. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@whoknows  Apache NiFi 2.0.0-M1 requires Java 21 and utilizes Jetty 10.  This results in needing to comply with the SNI specification.   So the URL used to access your NiFi can not use an IP address and the hostname used must match a hostname found in the SAN entries list with the NiFi's configured keystore PrivateKeyEntry. Apache NiFi 2.0 out-of-the-box will generate a keystore and truststore.  The keystore will contain a PrivateKeyEntry with a SAN entry for localhost and the server hostname. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Zifo1  Welcome to the community.... The Single User Authorizer is not a full featured authorization provider. It was only added to Apache NiFi so that out-of-the-box NiFi would be able to start securely easily for evaluation purposes.  It does not provide a mechanism for creating additional authorizations for other identities such as other NiFi instances. In order to support authorizing additional user/client identities against various NiFi policies, you'll need to switch to using a production ready authorizer like the "managed-authorizer". A typical example configuration would look like this: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#file-based-kerberos-authentication This setup uses the File-user-group-provider and file-access-policy-provider with the managed -authorizer. Now you could configure your single user provider identity as the "Initial User Identity 1" in the file-user-group-provider and as the "initial admin identity" in the file-access-policy-provider.  This would setup the needed admin policies for this user identity you need.   Note: Keep in mind that these providers will only generate the authorizations.xml and users.xml files the first time NiFi is started with this configuration. So if you set the above initial user and initial admin identities wrong, you'll need to fix config, delete these two files and start NiFi again so they are created again. Above does nothing with authentication since you are still using the single user authentication.  With this default authentication provider you can only authenticate with the single user identity or using a clientAuth certificates (which may also be challenging with default truststore your out-of the-box NiFi uses).   Authentication via a mutualTLS exchange is how Nifi node to node communications work and NiFi site-to-Site.  In order for mutual TLS exchange to be successful there must be mutual trust of the certificate exchanged.  So if one NiFi's certificate is not trusted by the other NiFi's truststore, authentication will not be possible.  So you may need to add additional trustedCertEntries (public certs) to both your NiFi's truststores before you'll be able to successfully negotiate the MutualTLS exchange/handshake. All the available authentication providers offered are documented here: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#user_authentication Aside from the single-user authentication provider, all other providers rely on some external source.  Apache NiFi does not offer a multi-user local authentication provider. I know this is a lot of info thus far, but should provide you the path to a slightly more production ready NiFi that will open up the ability to use additional features not available with the out-of-the-box setup. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Coordinador  You could upgrade your MongoDB to a newer version. Another option you may try is to add the Apache NiFi 1.16 *mongo* nars to your Apache NiFi 1.22 custom extension folder: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#autoloading-processors This will make both the 1.16 and 1.22 versions of the mongo processor available for use in your 1.22 Apache NiFi. The Mongo Driver has been updated numerous times between 1.16 and 1.23: https://issues.apache.org/jira/browse/NIFI-9886 https://issues.apache.org/jira/browse/NIFI-10557 https://issues.apache.org/jira/browse/NIFI-11419 https://issues.apache.org/jira/browse/NIFI-11856 If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@SAMSAL  The managed Authorizer uses the file-access-policy-provider (generates the authorizations.xml if it does no already exist) and then a user-group-provider.  In your case that would make most sense to be the ldap-user-group-provider.  You may also want to use the Composite-configurable-user-group-provider (configure it with ldap-user-group-provider and file-user-group-provider).  Having both a file based provider and ldap provider allows sycning of ldap users and groups  form ldap automatically as well as the file provider allowing you to manually add non ldap user/client identities for authorization as well.    Non ldap client/user identities might be certifcate based clients like other NiFi nodes/instance, etc.. Within the file-access-policy-provider you define the initial admin identity. That user identity could be set to your ldap user account identity.  Then on first start up with managed provider, it generates the authorizations.xml file seeded with the policies necessary for that initial admin user identity to act as admin.  So you could skip the single-user-provider step. Matt ... View more

@ChuckE  Plus 1 on @SAMSAL post.  I also had no problem with it working on my CFM 2.1.5.3 SP3 (Based off Apache NiFi 1.18 with many bug fixes from 1.19 - 1.20)      Thanks, Matt ... View more