@Rohit1997jio  Need to understand your use case more.... Agreed with @SAMSAL that the Wait and Notify processors are not what you want to use here. ControlRate processor may also not be your best option here.  assume you have multiple FlowFiles all route to failure.  All of those will reach that controlRate processor at about same time.  First goes right through if it has been 10 seconds since last Flowfile passed through it (so first is not delayed 10 seconds at all).  All FlowFiles queued behind allowing 1 to pass through every 10 seconds.  So with multiple FlowFiles you now have an even larger gap between processing. I see that you are using CFM 2.1.5 based off Apache NiFi 1.1.8.  In this version of NiFi the ability to handle retry on a relationship is built in to the processors framework.  This new feature is more powerful than the multiple processor method you are using now in the dataflow you shared above.  Your flow requires more processors and more resource usage. There is also no way to guarantee a exact 10 second delay.  So hope this is a soft requirement. Open the configuration for the PublishKafka processor and select the "Relationships" tab. On the "failure" relationship you have the ability to check the box for "retry".  when selected, additional configuration parameters appear.  Here you can specify the number of retry attempts to make before the FlowFile is routed to the "failure" relationship.  You also have the ability to control how this processor behaves when retry is happening: - Penalize (set/apply the penalty defined on the setting tab of processor to this FlowFile.  This FlowFile will not be retried until penaltly expires.  During this penalty duration processor will continue to process other FlowFiles.) - Yield (Same as penalize EXCEPT, no other FlowFiles will get processed until this FlowFile is sucessfully processed or routed to failure relationship.  Useful when processing order of FlowFiles i important). What this built in retry will not do is apply same penalty to every retry. Let's say you set "penalty duration" to 10 secs in setting tab.  Then you configure "number of retry attempts" to 3.  First failure results in FlowFile being penalized 10 secs, second attempt is penalized 20 secs, and third attempt is penalized 40 secs.   This incremental back-off help reduce resource usage when you have continuous failure (something else you flow will not do).  Once all attempts to retry are exhausted without success, FlowFile would rout to failure where you could use LogMessage and send failed FlowFile to different processing path like you did in your dataflow.  If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@edim2525  Apache NiFi is designed to move data fast and efficiently.  Each processor consumes FlowFiles from an incoming connection with no guaranteed priority/order, but depending on the processors used and concurrency setting, that does not guarantee order of FlowFiles execution.  Nor can order be guaranteed when FlowFiles are spread across nodes in a multi-node NiFi cluster. Now you did not mention if you are a single standalone instance of NiFi or a NiFi cluster.  You also do not mention if you have other NiFi component processors between your consumeKafka and PublishKafka processors. The ConsumeKafka processor consumes messages from a Topic and produces one FlowFile for each message.  For most efficient processing the ConsumeKafka processor would typically be have its concurrent tasks configured to equal or be less than number of partitions on the topic (never more as that causes constant rebalance).  This creates a consumer group allowing each concurrent thread execution to pull data from specific partitions assigned to the consumer group. As an example:  If you had a 3 partition topic and a 3 node NiFi cluster, you would configure your ConsumeKafka with 1 concurrent task (this is a total of 3 concurrent tasks across yoru 3 NiFi nodes). Each NiFi node would would be assigned a different topic partition to consume from.  Now each node in a NiFi cluster executes its own copy of the flow with no knowledge of or regards to what FlowFiles are be processed on other nodes.  This is why guaranteed order in a NiFi cluster is very difficult.  There are options to run some dataflows on Primary node only keeping that dataflow running on only one node.  But even then you can have the primary node election change and NiFi has no way to force or guarantee a specific nodes is assigned or retains the primary node role. Somethings you may try that might work for you... Suggestion 1: 1. Standalone NiFi instance (removes issues that a cluster can introduce when order is important) 2. When using ConsumeKafka, try using EnforceOrder processor after ConsumeKafka configured to order FlowFiles by the offset.  Then make sure that all connections downstream of EnforceOrder processor to PublishKafka use "FirstInFirstOutPrioritizer" (by default no prioritizer is set on a connection). 3. Make sure all processors in this dataflow are configured with only 1 concurrent task to avoid any concurrency that could lead to some FlowFile being processed faster then other impacting queued order in downstream connections. Suggestion 2: 1. Standalone NiFi instance for same reason as above. 2. Use ConsumeKafkaRecord instead of ConsumeKafka.  This produce fewer FlowFiles with each FlowFile containing multiple messages. 3. Use PublishKafkaRecord instead of PublishKafka  4. Make sure that all connections downstream of PublsihKafkaRecord processor to PublishKafkaRecord processor use "FirstInFirstOutPrioritizer" (by default no prioritizer is set on a connection). 5. still use one concurrent task on all processors to avoid concurrent record production. --- This option leads to fewer FlowFile (meaning less overhead and lower chance of messages becoming out of order) and likely faster throughput. Suggestion 3: If your dataflow consists of nothing more than consume and publish and you aer doing nothing with the data between these processor, as @steven-matison mentioned, NiFi may not be the best choice here.  NiFi is designed around faster and efficient movement of data using multi-node clusters  and concurrent thread execution.  Additionally NiFi needs to consume all the messages in to FlowFiles (writing content to content_repository and metatdata to the flowfile_repository) and  records data provenance which adds to overhead. So a purpose built tool for a simple replication task may be better for this use case. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@alexduta  NiFi's authorization policies control what users can and can't interact with within the NiFi UI.  If the user has been granted "modify the component" or "operate the component" policies on the processors or process group, the user will be able to initiate a start or stop of the process group. Of course the user needs "view the component" and "modify the component" on a process group in order to even add components to the process group.  Apache NiFi does not provide a way to disable start/stop from the process group level through any configuration file properties. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Knowledgeknow  You can't enable authentication and authorization on an unsecured NiFi (HTTP).  To enable security in NiFi, step one is to configure HTTPS (This will require you to have certificates for all your NiFi nodes).   The following configuration files have configurations related to securing your NiFi. nifi.properties --> (framework configuration file has bits related to authentication and authorization).  You would enable security on your NiFi by configuring HTTPS. Once NiFi is configured with an HTTPS port authentication via TLS certificates is enabled (Can NOT be disabled and is always first method attempted to authenticate a user/client).  Teh following section of this file pertain to security: Security Properties Identity Mapping Properties OpenID Connect - Since you mention Oauth2 and others... Login-identity-providers.xml --> (authentication related) used if you want to enable user authentication support through ldap or kerberos. To enable the ldap-provider or kerberos -provider, you'll need to specify one or the other in the nifi.properties configuration property: "nifi.security.user.login.identity.provider".  Out-of the-box NiFi has this configured to use the Single-User-Provider (not intended for production use). Lightweight Directory Access Protocol (LDAP) - Since you mention LDAP base authentication. Once you have decided on your authentication method of choice, you'll need to setup Multi-Tenant Authorization.  Authorization is used to control what your various successfully authenticated users/client have access to within NiFi's UI.  This gets configured in the authorizers.xml (order in which you add various providers to this configuration file is very important!!!).   This file consists of only one Authorizer (out of the box it uses the single-user-authorizer.  The "authorizer" is always at the very bottom of the authorizers.xml.file.   Below is a very common example structure (top to bottom order of providers added to file: FileUserGroupProvider LdapUserGroupProvider Composite Implementations - You'll want use "CompositeConfigurableUserGroupProvider". This is then configured to use both above UserGroupProviders. FileAccessPolicyProvider - configured to use above "composite-user-group-provider". StandardManagedAuthorizer - configured to use above "file-access-policy-provider" Example configuration of above:  https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#composite-file-and-ldap-based-usersgroups The authorizers.xml will setup the initial required properties for the user/client you define in the fileAccessPolicyProvider "Initial Admin Identity" (the user identity configured in this provider must be returned by ONLY one of the configured UserGroupProviders.  So do NOT configure the initial admin identity in the FileUserGroupProvider if that identity is going to be returned by the LDAPUserGroupProvider.  Don't worry if you mess up here initially, just delete the users.xml (FileUserGroupProvider generated) and authorizations.xml (FileAccessPolicyProvider generated) files and on next startup they will be created again. Once you have a working authentication and authorization setup, you will be able to define authorizations, using your InItial Admin user, for your other synced directly through the NiFi UI.  You can also define additional authorization for your admin user (is not given access to everything, but is given admin authorization which means this user can set new authorizations for all user including itself.  If you run it to authorization issue after setup, you'll want to inspect the nifi-user.log.  This log will show the exact case sensitive user/client identity. If it does not match exactly with the identity that was returned by the authorizer UserGroupProviders, you'll need to go back and make some configuration changes until they do.  Have fun in your journey.... If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Knowledgeknow  You can't enable authentication and authorization on an unsecured NiFi (HTTP).  To enable security in NiFi, step one is to configure HTTPS (This will require you to have certificates for all your NiFi nodes).   The following configuration files have configurations related to securing your NiFi. nifi.properties --> (framework configuration file has bits related to authentication and authorization).  You would enable security on your NiFi by configuring HTTPS. Once NiFi is configured with an HTTPS port authentication via TLS certificates is enabled (Can NOT be disabled and is always first method attempted to authenticate a user/client).  Teh following section of this file pertain to security: Security Properties Identity Mapping Properties OpenID Connect - Since you mention Oauth2 and others... Login-identity-providers.xml --> (authentication related) used if you want to enable user authentication support through ldap or kerberos. To enable the ldap-provider or kerberos -provider, you'll need to specify one or the other in the nifi.properties configuration property: "nifi.security.user.login.identity.provider".  Out-of the-box NiFi has this configured to use the Single-User-Provider (not intended for production use). Lightweight Directory Access Protocol (LDAP) - Since you mention LDAP base authentication. Once you have decided on your authentication method of choice, you'll need to setup Multi-Tenant Authorization.  Authorization is used to control what your various successfully authenticated users/client have access to within NiFi's UI.  This gets configured in the authorizers.xml (order in which you add various providers to this configuration file is very important!!!).   This file consists of only one Authorizer (out of the box it uses the single-user-authorizer.  The "authorizer" is always at the very bottom of the authorizers.xml.file.   Below is a very common example structure (top to bottom order of providers added to file: FileUserGroupProvider LdapUserGroupProvider Composite Implementations - You'll want use "CompositeConfigurableUserGroupProvider". This is then configured to use both above UserGroupProviders. FileAccessPolicyProvider - configured to use above "composite-user-group-provider". StandardManagedAuthorizer - configured to use above "file-access-policy-provider" Example configuration of above:  https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#composite-file-and-ldap-based-usersgroups The authorizers.xml will setup the initial required properties for the user/client you define in the fileAccessPolicyProvider "Initial Admin Identity" (the user identity configured in this provider must be returned by ONLY one of the configured UserGroupProviders.  So do NOT configure the initial admin identity in the FileUserGroupProvider if that identity is going to be returned by the LDAPUserGroupProvider.  Don't worry if you mess up here initially, just delete the users.xml (FileUserGroupProvider generated) and authorizations.xml (FileAccessPolicyProvider generated) files and on next startup they will be created again. Once you have a working authentication and authorization setup, you will be able to define authorizations, using your InItial Admin user, for your other synced directly through the NiFi UI.  You can also define additional authorization for your admin user (is not given access to everything, but is given admin authorization which means this user can set new authorizations for all user including itself.  If you run it to authorization issue after setup, you'll want to inspect the nifi-user.log.  This log will show the exact case sensitive user/client identity. If it does not match exactly with the identity that was returned by the authorizer UserGroupProviders, you'll need to go back and make some configuration changes until they do.  Have fun in your journey.... If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@ChuckE  You can right click on the UpdateAttribute processor and "view state" which will allow you to clear the stored state for that processor without needing to create a new processor. Every dynamic property you add becomes a different local state variable that is added to the FlowFiles as a FlowFile Attribute. Thanks, Matt ... View more

@Fayza  Apache CouchDB looks very limited in its query capabilities. But it does appear that a user in the community created nifi-couchDB components: https://github.com/wasperen/nifi-couchdb You could try adding this nar to your Apache NiFi installation to see if these components can satisfy your use case. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more