Created on 12-05-2018 03:18 PM - edited 08-17-2019 04:03 PM
We currently observe a strange behaviour with the LDAP integration for the NiFi Registry. I'm only able to create buckets using the initial admin, but can't add policies directly after starting the registry. All corresponding UI elements are inactive and it takes about 30 minutes until policies/users can finally be added.
I guess the sync of users and groups from our AD takes a lot of time and policies can't be added meanwhile. The logs show tons of messages indicating irrelevant groups are scanned.
2018-12-05 07:10:13,546 WARN [ (ldap-user-group-provider) - background sync thread] o.a.n.r.s.l.t.LdapUserGroupProvider [9899661b-5481-329c-b1a4-7b9b229c84dc] are members of CN=some-user-group,OU=ou1,OU=ou2,OU=ou3,OU=ou4,DC=somedomain,DC=local but that group was not found while searching users. Ignoring group membership.
Maybe it needs some adjustment to the search filters, however, the same settings in 'authorizers.xml' work without problems for NiFi itself.
<property name="User Search Base">ou=users,ou=ouxy,dc=domain,dc=local</property> <property name="User Object Class">person</property> <property name="User Search Scope">SUBTREE</property> <property name="User Search Filter">(memberOf=CN=main-nifi-user-group,OU=nifi-ou,OU=users,OU=ouxy,DC=domain,DC=local)</property> <property name="User Identity Attribute">sAMAccountName</property> <property name="User Group Name Attribute">memberOf</property> <property name="User Group Name Attribute - Referenced Group Attribute"></property> <property name="Group Search Base">ou=nifi-ou,ou=users,ou=ouxy,dc=domain,dc=local</property> <property name="Group Object Class">group</property> <property name="Group Search Scope">ONE_LEVEL</property> <property name="Group Search Filter"></property> <property name="Group Name Attribute">sAMAccountName</property> <property name="Group Member Attribute"></property> <property name="Group Member Attribute - Referenced User Attribute"></property>
Did anyone face similar issues? Any help would be highly appreciated.
Created 12-05-2018 04:22 PM
Update: Thought it would only occur on registry startup. However, corresponding UI elements are inaccessable on every login. Refreshing the browser window (F5) will make them usable again. Tested with various browsers on several clients with the same result.
Created 12-05-2018 07:24 PM
This is a known issue with NiFi and NiFi Registry when syncing users and groups from very large external directories. It just takes a long time. My recommendation would be the following:
Hope this helps,
Kevin
Created 12-06-2018 03:00 PM
Thank you very much for your suggestions, but 'User Search Base' and 'Group Search Base' are already set accordingly.
Unfortunately I don't see a way to further narrow down the syncing process without changing the structure of the AD itself. I followed your advice and increased the 'Sync Interval' to 120 minutes. Hopefully we won't face inaccessible UI elements too often with this setting.
However, I still got some questions...
It's not a real "show stopper" while we simply have to refresh the UI in order to make it working again. Just trying to understand how to optimize the LDAP syncronisation.
Thanks, Chris
Created 12-06-2018 05:28 PM
I do not think the long sync time period is caused by the group search. I think it is caused by the user search. You are correct that it looks like reducing the number of users to search to would require restructuring the AD. I understand that is not always possible / desire-able.
To answer your questions:
- Are you sure the NiFi and NiFi Registry settings are identical? What version of NiFi and NiFi Registry are you using? The LDAP sync procedure should be the same, so I would be surprised if there were very different behaviors.
- No, it is not possible. The algorithm basically pulls all users that match the user search settings then pulls all groups matching the group search settings, then iterates over the users/groups it has synced in order to build the mapping of users<->groups. Changing this sync algorithm would require a re-write of the code or implementing a custom LdapUserGroupProvider.
- It is not possible to specify two OUs for the User Search Base. You could fork the code and implement a modified/custom LdapUserGroupProvider that takes an array of search bases and does multiple searches (one for each base), but that capability is not something currently supported.
- For large directories, I've seen people have good luck specifying a page sizes of 100-500.
Kevin
Created 12-07-2018 02:56 PM
Thanks for the insight, @kdoran
Still trying to limit the amount of users somehow, but apparently its either restructuring the AD or writing a custom LdapUserGroupProvider. Both soloutions are out of scope unfortunately.
Yes, the settings are identical in NiFi and NiFi Registry. Actually I copied the 'authorizers.xml' from our NiFi instance and certainly didn't expect to run into the described issue. I compared both files and only the classes differ.
My conclusion was that something has to work differently with the registry's LDAP sync. I thought of upper-/lowercase, escape characters or other syntax problems, but the docs didn't provide any clues.
We used the official docker images for our deployment, so we are on NiFi 1.8.0 and NiFi Registry 0.3.0 right now.
Will look into the page size property as well. Thanks.
Chris
Created 12-06-2018 03:00 PM
Thank you very much for your suggestions, but 'User Search Base' and 'Group Search Base' are already set accordingly.
Unfortunately I don't see a way to further narrow down the syncing process without changing the structure of the AD itself. I followed your advice and increased the 'Sync Interval' to 120 minutes. Hopefully we won't face inaccessible UI elements too often with this setting.
However, I still got some questions...
It's not a real "show stopper" while we simply have to refresh the UI in order to make it working again. Just trying to understand how to optimize the LDAP syncronisation.
Thanks, Chris