@Frank168  I'd expect what you are seeing with the long sync times.  Sync interval has not impact in speed of the user an group syncs.  This simply controls how often NiFi is expected re-sync with ldap.  So in your case, NiFi would finding one sync and immediately start the next.  NiFi does a sync during startup as it loads the authorizers.xml, NiFi does not continue to load until that initial sync completes otherwise it can't do authorizations.   After that initial successful sync with ldap, NiFi will finish loading and only then will UI become available.   While running the sync interval will run in the background to re-sync from ldap in case any users or groups are added/removed/updated since last sync.    some users adjust this to sync every 24 hours instead of every 30 minutes.   This depends on how quickly you want NiFi to be aware of changes. As I commented before, you are loading 50,000+ users into NiFi heap memory (this impact the free heap available to the NiFi process for executing your dataflows on the canvas) and then redoing that every sync interval.  This was not how NiFi ldap-user-group-provider was intended to be used.   The only users and groups that should be loaded in to NiFi are those you plan to establish authorization policies for and will be accessing your secured NiFi. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Krish98  Unfortunately, I have no "Best Practice" recommendation for the use case you have shared.  It is not a use case I have ever setup before. Also want to share that in Apache NiFi 2.4+ version a new JWTBearerOAuth2AccessTokenProvider controller service was introduced.  While not a solution to you query, I wanted to share this with you.  Apache NiFi jira: NIFI-14380 NOTE: The Apache NiFi 1.x major release line is End-Of-Life now.  There will be no future releases of the 1. major release line.  There is no direct upgrade path from Apache NiFi 1.x to Apache NiFi 2.x.  You'll need to migrate your dataflows from 1.x to 2.x. For our Cloudera Flow Management licensed users, we provide tooling to assist with migrating dataflows from Flow Management versions based on Apache NiFi 1.x to Flow Management versions based on Apache NiFi 2.x.   Cloudera Flow Management 2.x also includes many of the components deprecated and no longer included in the Apache NiFi 2.x release line.  Thank you, Matt ... View more

@Frank168  When NiFi fails to start, you'll see logging in the nifi-app.log with details as to why.  My guess would be it failed when loading the authorizer because the ldap-user-group-provider execution was unsuccessful.  That in turn was likely caused because you set the client page size to a value higher then the max enforced by the ldap server.  LDAP server common MaxPageSize settings are 500 or 1000.   Setting Page Size in NiFi to 500 is a common practice.  The NiFi ldap client will request all returns in pages of 500.  The Ldap server will return as many pages as required to fulfill the request.  When page size is left blank in NiFi, the ldap server is only going to return one page limiting the result set to whatever fit in the ldap server page page size which would explain your missing returns. Asa reminder, I caution against returning so  many users.    Do you expect all 1700+ users to be accessing your NiFi?   That would be extremely uncommon.   Typical usage is that ldap users that will be accessing NiFi are added to a few specific ldap groups. Then the ldap-user-group-provider is configured to fetch only those groups and group members instead of syncing everyone from ldap.  This strategy limits the heap consumed by the return user and group set.  Keep in mind that the ldap-user-group-provider default is to re-sync every 30 minutes also.  Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Frank168  What you expect is exactly how it works.  Did you configure any user search based filtering in your LdapuserGroupProvider?  Is your Referral Strategy set to FOLLOW?  Any exceptions in the nifi-user.log or nifi-app.log?  How many users are being returned successfully (count)? Did you set your Page Size? Ldap often limits the number of returns in a single page.  NiFi by default does not set a "Page Size" so it request all results in one page.  So depending on how many users you are trying to return and your ldap's defaults, this could result in missing users.  Try setting the "Page Size" in NiFi to 500 to see if that resolves your issue. NOTE:  All user and group identities returned by the ldap-user-group-provider are held in NiFi's heap memory.   You should avoid syncing very large sets of users and groups and utilize user and group search filtering to only return those users and groups that will be accessing NiFi. If page size is not your issue, inspecting your setup may provide other clues.   Can you share your login-identity-providers.xml and authorizers.xml file configurations (redact sensitive values)? Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@OfekRo1  This is a very old community question.  It would be best for you to start a new community question with your setup specific (Apache NiFi version, Target JMS vendor and version, attempted ConsumeJMS / PublishJMS processor configuration, Current JMSConnectionFactoryProvider configuration).    The Additional Details...  section of the JMSCOnnectionFactoryProvider may also be helpful to you.   Sharing any related logging from your nifi-app.log relative to these components when attempting to run the dataflow in your NiFi may be helpful when you create your new community question. Thank you, Matt ... View more

@hckorkmaz01  @vafs provided a couple excellent options already. As a third option, you could utilize the SiteToSiteStatusReportingTask.  You can configure this NiFi reporting task to send metrics for only connections and you can add a regex to limited the number of connections returned (this requires renaming the desired connections so you have more granular control with the regex).  This Reporting task can be configured to send to a remote input port on on the same NiFi.   This Reporting task will execute the query on each host in your NiFi Cluster and send results to the the desired Site-To-Site remote input port.   So in a three node NiFi cluster three FlowFiles will be produced and sent with Json content for connection(s) you filtered on.   This way you can see what is queued per node.  Limitations to monitoring connections in this way exist.  Keep in mind that you are grabbing a snapshot at one specific moment in time.  So if the downstream component of this monitored connection is running, the connection stat would have likely changed by the time you got the last status details.   Now if you are using this to monitor flow dead-ends where you don't expect FlowFiles to ever accumulate, that is perhaps a valid use case. Example json output placed in FlowFile: [ { "statusId" : "27b01368-8280-4e5e-ac22-0e749c46fe17", "timestampMillis" : 1771336855410, "timestamp" : "2026-02-17T14:00:55.410Z", "actorHostname" : "nifi-node-1", "componentType" : "Connection", "componentName" : "Monitored-success", "parentId" : "603017c1-0197-1000-0000-000013171e7c", "parentName" : "test", "parentPath" : "NiFi Flow / test", "platform" : "nifi", "application" : "NiFi Flow", "componentId" : "508a2293-019a-1000-ffff-fffff15dc582", "sourceId" : "5089d4ab-019a-1000-ffff-ffff9b682ab3", "sourceName" : "UpdateAttribute", "destinationId" : "4dc23b89-25b0-1eff-b974-6fe5425c283f", "destinationName" : "UpdateAttribute", "maxQueuedBytes" : 0, "maxQueuedCount" : 0, "queuedBytes" : 30720, "queuedCount" : 30, "inputBytes" : 0, "inputCount" : 0, "outputBytes" : 0, "outputCount" : 0, "backPressureBytesThreshold" : 1073741824, "backPressureObjectThreshold" : 10000, "backPressureDataSizeThreshold" : "1 GB", "isBackPressureEnabled" : "false" } ]   IMPORTANT NOTE: I'd also like to point out that Apache NiFi 1.18 was released back in 2021 and newer releases have addressed many bugs and CVEs.  I strongly encourage you to transition to a newer version of Apache NiFi.  If you aren't ready to migrate to the newer Apache NiFi 2.x major release versions, you can still easily upgrade to last Apache NiFi 1.x major release version (1.28.0). Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@william_gosse @IgorSpryzhkov  A more modern update to this Community question. The original post references guidance given for a List<type> processor to corresponding fetch<type> processor model that redistributes the FlowFile created by the list<type> processor running on the elected primary node only to all nodes so that all nodes share the heavy work of fetching the file contents and processing that data.  NiFi now offers a much simpler approach via the load balanced strategy option.   This removes the complexity of the Remote Process Group setup.  Simply connect your List<type> processor success relationship directly to the corresponding Fetch<Type> processor via a connection.  Then edit the configuration of that connection to enable load balancing.  There are four options to choose from but in this scenario the  "Round Robin" options is what you would use most commonly.    Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@zzzz77  Provenance can be very noisy depending on size of your dataflows and the amount of FlowFIles being processed through those dataflows.  The provenance repo has age and size configuration that trigger roll-off of old events.   So you may not reach the retention age if you reach size first.  Also would not be trying to read provenance files while they are being written to.    The SiteToSiteProvenanceReportingTask might be the solution you are looking for in Apache NiFi.   This reporting task will send all provenance events over Site-To-Site protocol to a target NiFi where you can then feed them into any long term storage medium of your choice in a human readable format.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@zzzz77   I can certainly help you with the structured setup commonly used when integrating NIFi with LDAP.  NiFi authentication and authorization are different processes and configurations.  You can even authenticate using LDAP and not use LDAP at all during authorization. Also need to be aware that only a secured NiFi setup over HTTPS can support authentication and authorization. Since Authentication needs to happen first, we'll start there. LDAP authentication is configured as a login provider inside the login-identity-providers.xml configuration file:   <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Authentication Strategy">START_TLS</property> <property name="Manager DN"></property> <property name="Manager Password"></property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url"></property> <property name="User Search Base"></property> <property name="User Search Filter"></property> <property name="Identity Strategy">USE_DN</property> <property name="Authentication Expiration">12 hours</property> </provider>   The actual configuration is dependent on your LDAP setup.  You can refer to the linked documentation for each field.  Depending on "Authentication Strategy" setting, TLS properties may not need to be configured.  The "identifier" for this provider is "ldap-provider".   The "Identity Strategy" is used to decide what string is used as the authenticated users identity.  Options are "USE_DN" (use the full DN from the LDAP entry) or "USE_USERNAME" (use the username as typed in the login window).   USE_USERNAME is commonly used. This identifier needs to be configured in the nifi.properties file, so NiFi knows which login-provider NiFi should be using.    nifi.security.user.login.identity.provider=ldap-provider     Now we need to setup the authorizers.xml file so we can setup authorizations for the ldap users.  Here you have two options, you can manually add the ldap user identities via the "user-group-provider" or you can sync the user identities directly from ldap using the "ldap-user-group-provider".  Sometimes you want both if not all your users/clients are part of LDAP (this applies to user identities derived from clientAuth certificates during a mutualTLS exchange).   Both would commonly be necessary for a NiFi cluster setup.   Since you are setting up a single instance (non cluster) NiFi, I'll show how to structure your authorizers.xml file using just the ldap-user-group-provider: <userGroupProvider> <identifier>ldap-user-group-provider</identifier> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">cn=Manager,dc=nifi,dc=hwx</property> <property name="Manager Password">password</property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://<ip or hostname>:389</property> <property name="Page Size">500</property> <property name="Sync Interval">30 mins</property> <property name="Group Membership - Enforce Case Sensitivity">false</property> <property name="User Search Base">ou=People,dc=nifi,dc=hwx</property> <property name="User Object Class">inetOrgPerson</property> <property name="User Search Scope">SUBTREE</property> <property name="User Search Filter"></property> <property name="User Identity Attribute">cn</property> <property name="User Group Name Attribute">memberOf</property> <property name="User Group Name Attribute - Referenced Group Attribute"></property> <property name="Group Search Base">ou=Group,dc=nifi,dc=hwx</property> <property name="Group Object Class">groupOfNames</property> <property name="Group Search Scope">SUBTREE</property> <property name="Group Search Filter"></property> <property name="Group Name Attribute">cn</property> <property name="Group Member Attribute">member</property> <property name="Group Member Attribute - Referenced User Attribute"></property> </userGroupProvider> <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> <property name="User Group Provider">ldap-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">nifiadmin</property> <property name="Node Identity 1"></property> <property name="Node Group"></property> </accessPolicyProvider> <authorizer> <identifier>managed-authorizer</identifier> <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class> <property name="Access Policy Provider">file-access-policy-provider</property> </authorizer> Above authorizer is the most basic setup example assuming an unsecure ldap setup as the example. You can see it has three sections.  The bets way to read an authorizers.xml configuration is from the bottom up starting with the "authorizer".  In this example you can see I am using the "StandardManagedAuthorizer" which has an identifier of "managed-authorizer" and it is configured to reference the "file-access-policy-provider".  So the next provider we should find going up through the authorizers.xml will be the provider with the identifier "file-access-policy-provider". The "FileAccessPolicyProvider" is responsible for persisting the granted authorizations in a file name "authorizations.xml". This provider will also set some initial authorizations for the user identity set in the "Initial Admin Identity" field and the for any "Node Identity <num>" field entries.   We can see that this provider is learning about users and groups from the "ldap-user-group-provider". IMPORTANT NOTES:  This provider will only create the authorizations.xml file if it does NOT already exist.  So if you make any changes to this provider, those changes would not be reflected in an already existing authorizations.xml file.   Also any identity strings set this provider must be returned by a user-group-provider(s).  So the next provider needed has the identifier "ldap-user-group-provider" and needs to be located further up in this authorizations.xml file.  So we locate the "LdapUserGroupProvider" which has this identifier.  This provider has no reference to any additional providers.  While i shared a very basic sample configuration, your configuration will be specific to your ldap server source. My example is configured to sync users and groups from ldap.  You can choose to sync users or users and groups.  You can not sync just groups.   Inside the nifi.properties file you will set the authorizer you want to use: nifi.security.user.authorizer=managed-authorizer Now that we have the authentication and authorization setup complete, let's walk through what happens when you access NiFi's "https://<hostname>:<port>/nifi" url. A mutualTLS exchange with the client (browser) will occur where NiFi will "WANT" a clientAuth certificate.  Of one is not presented in that exchange, NiFi will redirect to the login UI: Here the user will supply their ldap username and password.   Assuming the ldap-login-identity-provider is using "USE_USERNAME" and authentication was successful, the username (case sensitive) as typed in the username field will be passed to the managed authorizer to check what authorizations are in place for that user.  Before that user identity reaches the managed authorizer, it is compared against the any Identity Mapping Properties configured in the nifi.properties file to see if any string manipulation should happen.  Next the string (manipulated if mapping was applied) goes to the authorizer.  First the authorizer will check to see if that user identity belongs to any groups.  Then it will check if the user or any groups that user is known to be member of (based on returns from ldap-user-group-provider sync) has proper authorizations to access the NiFi UI.   If proper authorization exist, you will see the NiFi UI and the user identity will show in the upper right corner. If there are authorization issues, you'll find that logged in the nifi-user.log.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more