User Search Base in NiFi LDAP

[#1]

I am trying to setup LDAP in NiFi (v2.5.0) , and run into issues that it doesnt show all users from sub-ous,

User Search Base: ou=Users,dc=example,dc=com, and User Search Scope is SUBTREE
It shows some users, but not all.
If I change the User search base to the exact sub OU, and then it will show the expected users, e.g.  ou=XYZ,ou=ABC,ou=Users,dc=example,dc=com

I thought by specifying the user search base at a higher level, with substree as the search scope, it should list all users below?

Can someone advise anything else I should be changing in the LDAP User/Group provider settings to get this working?

[#2]

@Frank168 

What you expect is exactly how it works.  Did you configure any user search based filtering in your LdapuserGroupProvider?  Is your Referral Strategy set to FOLLOW?  Any exceptions in the nifi-user.log or nifi-app.log? 

How many users are being returned successfully (count)? Did you set your Page Size? Ldap often limits the number of returns in a single page.  NiFi by default does not set a "Page Size" so it request all results in one page.  So depending on how many users you are trying to return and your ldap's defaults, this could result in missing users.  Try setting the "Page Size" in NiFi to 500 to see if that resolves your issue.

NOTE:  All user and group identities returned by the ldap-user-group-provider are held in NiFi's heap memory.   You should avoid syncing very large sets of users and groups and utilize user and group search filtering to only return those users and groups that will be accessing NiFi.

If page size is not your issue, inspecting your setup may provide other clues.  
Can you share your login-identity-providers.xml and authorizers.xml file configurations (redact sensitive values)?

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3]

Thanks Matt. The user search filter is set with this value: (sAMAccountName={0})
and yes, Referral Strategy set to FOLLOW. No exceptions in the user log.
When I set the base dn to a parent OU, it returns roughly 1700 users.
I did try to set a page size, but when I do that, NiFi fails to start. Let me try again and set it to a smaller number, and see if works.

[#4]

@Frank168 

When NiFi fails to start, you'll see logging in the nifi-app.log with details as to why.  My guess would be it failed when loading the authorizer because the ldap-user-group-provider execution was unsuccessful.  That in turn was likely caused because you set the client page size to a value higher then the max enforced by the ldap server.  LDAP server common MaxPageSize settings are 500 or 1000.   Setting Page Size in NiFi to 500 is a common practice.  The NiFi ldap client will request all returns in pages of 500.  The Ldap server will return as many pages as required to fulfill the request.  When page size is left blank in NiFi, the ldap server is only going to return one page limiting the result set to whatever fit in the ldap server page page size which would explain your missing returns.

Asa reminder, I caution against returning so  many users.    Do you expect all 1700+ users to be accessing your NiFi?   That would be extremely uncommon.   Typical usage is that ldap users that will be accessing NiFi are added to a few specific ldap groups. Then the ldap-user-group-provider is configured to fetch only those groups and group members instead of syncing everyone from ldap.  This strategy limits the heap consumed by the return user and group set.  Keep in mind that the ldap-user-group-provider default is to re-sync every 30 minutes also. 

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#5]

Ok, after further testing, it works with the page size set at 300 when I selected the OU with just under 4000 users. If I change the user search base to a parent OU with much more users e.g. 50000,  it didnt seem to work no matter what page size I set. And logs not showing any errors. However it was actually still loading I think, but NiFi is not accessible during that period. What I found out is that after 30 - 35 minutes, NiFi is accessible. Not sure if its the size or something else, if I change the sync interval from 30 minutes to 10 minutes, it still took at least 30 minutes to make NiFi accessible.
Taken your point, will be using the user search filter (only if they are a member of particular AD groups) to manage the expected users to be loaded in NiFi.

[#6]

@Frank168 

I'd expect what you are seeing with the long sync times.  Sync interval has not impact in speed of the user an group syncs.  This simply controls how often NiFi is expected re-sync with ldap.  So in your case, NiFi would finding one sync and immediately start the next.  NiFi does a sync during startup as it loads the authorizers.xml, NiFi does not continue to load until that initial sync completes otherwise it can't do authorizations.   After that initial successful sync with ldap, NiFi will finish loading and only then will UI become available.   While running the sync interval will run in the background to re-sync from ldap in case any users or groups are added/removed/updated since last sync.    some users adjust this to sync every 24 hours instead of every 30 minutes.   This depends on how quickly you want NiFi to be aware of changes.

As I commented before, you are loading 50,000+ users into NiFi heap memory (this impact the free heap available to the NiFi process for executing your dataflows on the canvas) and then redoing that every sync interval.  This was not how NiFi ldap-user-group-provider was intended to be used.   The only users and groups that should be loaded in to NiFi are those you plan to establish authorization policies for and will be accessing your secured NiFi.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt