@zzzz77   I can certainly help you with the structured setup commonly used when integrating NIFi with LDAP.  NiFi authentication and authorization are different processes and configurations.  You can even authenticate using LDAP and not use LDAP at all during authorization. Also need to be aware that only a secured NiFi setup over HTTPS can support authentication and authorization. Since Authentication needs to happen first, we'll start there. LDAP authentication is configured as a login provider inside the login-identity-providers.xml configuration file:   <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Authentication Strategy">START_TLS</property> <property name="Manager DN"></property> <property name="Manager Password"></property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url"></property> <property name="User Search Base"></property> <property name="User Search Filter"></property> <property name="Identity Strategy">USE_DN</property> <property name="Authentication Expiration">12 hours</property> </provider>   The actual configuration is dependent on your LDAP setup.  You can refer to the linked documentation for each field.  Depending on "Authentication Strategy" setting, TLS properties may not need to be configured.  The "identifier" for this provider is "ldap-provider".   The "Identity Strategy" is used to decide what string is used as the authenticated users identity.  Options are "USE_DN" (use the full DN from the LDAP entry) or "USE_USERNAME" (use the username as typed in the login window).   USE_USERNAME is commonly used. This identifier needs to be configured in the nifi.properties file, so NiFi knows which login-provider NiFi should be using.    nifi.security.user.login.identity.provider=ldap-provider     Now we need to setup the authorizers.xml file so we can setup authorizations for the ldap users.  Here you have two options, you can manually add the ldap user identities via the "user-group-provider" or you can sync the user identities directly from ldap using the "ldap-user-group-provider".  Sometimes you want both if not all your users/clients are part of LDAP (this applies to user identities derived from clientAuth certificates during a mutualTLS exchange).   Both would commonly be necessary for a NiFi cluster setup.   Since you are setting up a single instance (non cluster) NiFi, I'll show how to structure your authorizers.xml file using just the ldap-user-group-provider: <userGroupProvider> <identifier>ldap-user-group-provider</identifier> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">cn=Manager,dc=nifi,dc=hwx</property> <property name="Manager Password">password</property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://<ip or hostname>:389</property> <property name="Page Size">500</property> <property name="Sync Interval">30 mins</property> <property name="Group Membership - Enforce Case Sensitivity">false</property> <property name="User Search Base">ou=People,dc=nifi,dc=hwx</property> <property name="User Object Class">inetOrgPerson</property> <property name="User Search Scope">SUBTREE</property> <property name="User Search Filter"></property> <property name="User Identity Attribute">cn</property> <property name="User Group Name Attribute">memberOf</property> <property name="User Group Name Attribute - Referenced Group Attribute"></property> <property name="Group Search Base">ou=Group,dc=nifi,dc=hwx</property> <property name="Group Object Class">groupOfNames</property> <property name="Group Search Scope">SUBTREE</property> <property name="Group Search Filter"></property> <property name="Group Name Attribute">cn</property> <property name="Group Member Attribute">member</property> <property name="Group Member Attribute - Referenced User Attribute"></property> </userGroupProvider> <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> <property name="User Group Provider">ldap-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">nifiadmin</property> <property name="Node Identity 1"></property> <property name="Node Group"></property> </accessPolicyProvider> <authorizer> <identifier>managed-authorizer</identifier> <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class> <property name="Access Policy Provider">file-access-policy-provider</property> </authorizer> Above authorizer is the most basic setup example assuming an unsecure ldap setup as the example. You can see it has three sections.  The bets way to read an authorizers.xml configuration is from the bottom up starting with the "authorizer".  In this example you can see I am using the "StandardManagedAuthorizer" which has an identifier of "managed-authorizer" and it is configured to reference the "file-access-policy-provider".  So the next provider we should find going up through the authorizers.xml will be the provider with the identifier "file-access-policy-provider". The "FileAccessPolicyProvider" is responsible for persisting the granted authorizations in a file name "authorizations.xml". This provider will also set some initial authorizations for the user identity set in the "Initial Admin Identity" field and the for any "Node Identity <num>" field entries.   We can see that this provider is learning about users and groups from the "ldap-user-group-provider". IMPORTANT NOTES:  This provider will only create the authorizations.xml file if it does NOT already exist.  So if you make any changes to this provider, those changes would not be reflected in an already existing authorizations.xml file.   Also any identity strings set this provider must be returned by a user-group-provider(s).  So the next provider needed has the identifier "ldap-user-group-provider" and needs to be located further up in this authorizations.xml file.  So we locate the "LdapUserGroupProvider" which has this identifier.  This provider has no reference to any additional providers.  While i shared a very basic sample configuration, your configuration will be specific to your ldap server source. My example is configured to sync users and groups from ldap.  You can choose to sync users or users and groups.  You can not sync just groups.   Inside the nifi.properties file you will set the authorizer you want to use: nifi.security.user.authorizer=managed-authorizer Now that we have the authentication and authorization setup complete, let's walk through what happens when you access NiFi's "https://<hostname>:<port>/nifi" url. A mutualTLS exchange with the client (browser) will occur where NiFi will "WANT" a clientAuth certificate.  Of one is not presented in that exchange, NiFi will redirect to the login UI: Here the user will supply their ldap username and password.   Assuming the ldap-login-identity-provider is using "USE_USERNAME" and authentication was successful, the username (case sensitive) as typed in the username field will be passed to the managed authorizer to check what authorizations are in place for that user.  Before that user identity reaches the managed authorizer, it is compared against the any Identity Mapping Properties configured in the nifi.properties file to see if any string manipulation should happen.  Next the string (manipulated if mapping was applied) goes to the authorizer.  First the authorizer will check to see if that user identity belongs to any groups.  Then it will check if the user or any groups that user is known to be member of (based on returns from ldap-user-group-provider sync) has proper authorizations to access the NiFi UI.   If proper authorization exist, you will see the NiFi UI and the user identity will show in the upper right corner. If there are authorization issues, you'll find that logged in the nifi-user.log.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@fy-test  I can't speak to your specific Zookeeper setup.  However, from a NiFi standpoint... NiFi-Registry has not dependency on Zookeeper, so it can be started at anytime. NiFi cluster setups have a requirement for zookeeper quorum before the NiFi cluster can be formed.  NiFi cluster can be started even without ZK quorum, but all nodes will be in a disconnected state until the ZK quorum is established and one of the NiFi cluster nodes is elected as the cluster coordinator by ZK, at which time all nodes will start sending heartbeats to that elected cluster coordinator and the cluster will be formed/established.  Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Frank168  Unfortunately, Apache NiFi does not support Nested Groups. There is an existing Apache NiFi Jira (NIFI-8035) for such an improvement, but it has never been implemented.  The existing implementation of the ldap-user-group-provider would treat all members of a group as users and does not validate the type of member.  Any change here would require NiFi to retrieve the object class of all members of a group and then conduct another search of any that were of identified as a group to retrieve their members and so on until all users are identified throughout the entire nested group tree.    Something to keep in mind here is that all the user and group identities along with associations are held in the NiFi heap memory on every node.  So doing such could result in a lot of user and groups consuming NiFi heap memory.   You should configure your Ldap-user-group-provider to sync only the groups from which users exist that will be accessing your NiFi limiting the length of time it takes to sync every 30 minutes and the heap memory impact.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@fy-test  Starting only on ZK node will not give quorum, so the NiFi cluster would not form.   NiFi nodes would come up and continue to attempt to connect to ZK quorum for cluster coordinator election before cluster could form. All NiFi nodes need to learn which node is elected to this role in order to know which node to send heartbeats to in order to form a NiFi cluster.  I'd say you have some other issues if your ZK quorum cluster is not stable when NiFi is started.  My ZK is completely up with quorum when I start any of my NiFi clusters.  If Quorum keeps coming and going due to some issue in your ZK, that could cause NiFi nodes to disconnect from cluster and reconnect when quorum exists again.  The real question here is why is your ZK not coming up well when you start all of the ZK hosts at the same time.  I'd spend more time looking at the health of your ZK. All you should need to do is start ZK nodes so you have quorum and then start NiFi and NiFi-Registry (order of NiFi and NiFi-Registry start does not matter).   Thanks, Matt ... View more

@hegdemahendra  Did you take heap dumps to confirm which class was consuming the heap? Any thread dump analysis when heap usage was growing? What incorrect values were configured that you feel led to this component consuming large amounts of heap memory? Not really finding any known issues of memory with this consumeKafka processor.  Any details you can provide (processor configuration, log exceptions while it was running with bad config, etc) may help. Thank you, Matt ... View more

@fy-test  I would not expect this step to be necessary: Delete flow.json.gz on disconnected nodes → successful rejoin - Flow election happens during startup. Once a flow is elected, nodes that join afterwards will inherit the cluster flow if their local flow does not match.   I see no need to clear ZK state.   ZK elects a cluster coordinator and primary node from the nodes that establish connection with ZK.   ZK also used for components in your dataflows that utilize cluster state. Clearing ZK could result in duplicate data processing depending in what your flow does in NiFi. This is interesting line shared from your logs. no heartbeat from node in 15089 seconds This implies that the elected cluster coordinator disconnected a node after not receiving a heartbeat for 15089 seconds.  This means the node was in a connected state.  On startup of a cluster, all nodes are in a disconnected state initially until they connect, so this is not a line I would expect to see during startup.  Was this accompanied by a line that stated node was being disconnected due to lack of heartbeat? Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@fy-test  Apache NiFi 2.7.2 does not use a flow.xml.gz file (This format was only used by Apache NiFi 1.x versions).   Apache NiFi 2.x versions use a flow.json.gz format.  I would suggest making sure the Zookeeper quorum is up before starting the NiFi Service. NiFi cluster can't form or remain formed if Zookeeper does not have stable quorum. If your NiFi nodes are disconnecting and reconnecting, I would start by looking at the status of the nodes to see what reason is being given for the disconnects.   You can find this in the Cluster UI within NiFi: Clicking on the small "i" icon to the left of the node name will open the pop-up window above that shows node events.  You should also see node events in the nifi-app.logs on each node.  So you see your elected cluster coordinator constantly changing? Was there a duration of time between stop and start? Was there a large influx of backlogged data when NiFi was started? Encounter any OutOfMemory exceptions? Encounter any long garbage collection events? Did nifi-app.log on elected cluster coordinator(s) reported any node disconnected due to lack of heartbeat log output? You would normally start all nodes at the same time.  NiFi knows how many nodes were last in the cluster and has a flow election process that depends on all nodes connecting. So startup times will be mush longer if not all nodes connect.  NiFi has a configurable timer of how long flow election will run before finishing starting with just the nodes that connected.  Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@zzzz77  FlowFile Metadata/attributes are held in NiFi Heap memory.  For queued FlowFiles, there is a configurable swap threshold in the nifi.properties that will swap batches of 10,000 FlowFIle's worth for metadata/attributes to disk when the threshold is met.  This swapping is there to minimize excessive heap usage when queues grow large.  The NiFi Content is not held in heap memory; however, some processor may need to read the content into heap memory for the processor to perform it's function.    You will notice if you look at the individual components documentation that a "System Resource Considerations" section exists.  If Heap memory usage is a concern for that processor, it will be documented there.  SplitContent processor docs example: Processors like SplitContent will hold the all the FlowFile metadata/attributes (not content) for every split FlowFIle being produced in heap memory until all the output FlowFiles have been produced and committed to the downstream connection.  These FlowFiles being produced can not be swapped to disk until they committed to the downstream connection.  So if a splitContent were to produce 50,000 split FlowFiles, the attributes for all 50,000 would be held in heap.  After committed to the downstream connection. 40,000 of those would get swapped to disk based on default swap thresholds. So heap impact would spike but not persist.  Since you have not shared the specific of your dataflow in question (which processors you are using), I can't provide any specific feedback.   Where is the chunking and de-chunking happening?  Sounds like this may be happening at source and at destination. NiFi is just moving these chunks from source to destination.  How are you sending the chunks to NiFi and transferring them to destination?   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Runa27  Without details of your database table structure/configuration and your test file, it would be challenging to identify your exact issue. Have you tried setting the "Unmatched Column Behavior" property to "Ignore Unmatched Columns" or "Warn on Unmatched Columns" to see if that makes a difference? Can you share how your CSVReader has been configured? Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt       ... View more