Initially unable to create policies in NiFi Registry

[#1]

We currently observe a strange behaviour with the LDAP integration for the NiFi Registry. I'm only able to create buckets using the initial admin, but can't add policies directly after starting the registry. All corresponding UI elements are inactive and it takes about 30 minutes until policies/users can finally be added.

I guess the sync of users and groups from our AD takes a lot of time and policies can't be added meanwhile. The logs show tons of messages indicating irrelevant groups are scanned.

2018-12-05 07:10:13,546 WARN [ (ldap-user-group-provider) - background sync thread] o.a.n.r.s.l.t.LdapUserGroupProvider [9899661b-5481-329c-b1a4-7b9b229c84dc] are members of CN=some-user-group,OU=ou1,OU=ou2,OU=ou3,OU=ou4,DC=somedomain,DC=local but that group was not found while searching users. Ignoring group membership.

Maybe it needs some adjustment to the search filters, however, the same settings in 'authorizers.xml' work without problems for NiFi itself.

<property name="User Search Base">ou=users,ou=ouxy,dc=domain,dc=local</property>
<property name="User Object Class">person</property>
<property name="User Search Scope">SUBTREE</property>
<property name="User Search Filter">(memberOf=CN=main-nifi-user-group,OU=nifi-ou,OU=users,OU=ouxy,DC=domain,DC=local)</property>
<property name="User Identity Attribute">sAMAccountName</property>
<property name="User Group Name Attribute">memberOf</property>
<property name="User Group Name Attribute - Referenced Group Attribute"></property>
<property name="Group Search Base">ou=nifi-ou,ou=users,ou=ouxy,dc=domain,dc=local</property>
<property name="Group Object Class">group</property>
<property name="Group Search Scope">ONE_LEVEL</property>
<property name="Group Search Filter"></property>
<property name="Group Name Attribute">sAMAccountName</property>
<property name="Group Member Attribute"></property>
<property name="Group Member Attribute - Referenced User Attribute"></property>

Did anyone face similar issues? Any help would be highly appreciated.

[#2]

Update: Thought it would only occur on registry startup. However, corresponding UI elements are inaccessable on every login. Refreshing the browser window (F5) will make them usable again. Tested with various browsers on several clients with the same result.

[#3]

@Christian Ludwig

This is a known issue with NiFi and NiFi Registry when syncing users and groups from very large external directories. It just takes a long time. My recommendation would be the following:

1. Limit the User Search Base and Group Search Base to be as tight as possible, and use the User Search Filter and Group Search Filter to further limit the number of users and groups that need to be synced.
2. Tune the LdapUserGroupProvider 'Page Size' and 'Sync Interval' properties.

Hope this helps,
Kevin

[#4]

@kdoran

Thank you very much for your suggestions, but 'User Search Base' and 'Group Search Base' are already set accordingly.

• We do have a specific OU containing the NiFi relevant groups. This is were the 'Group Search Base' points to and it only contains two groups.
• It is necessary to define the base OU for users, because there is no specific OU for NiFi relevant users. Therefore the 'User Search Filter' is set to filter only members of a single group (CN=main-nifi-user-group).

Unfortunately I don't see a way to further narrow down the syncing process without changing the structure of the AD itself. I followed your advice and increased the 'Sync Interval' to 120 minutes. Hopefully we won't face inaccessible UI elements too often with this setting.

However, I still got some questions...

1. Why does NiFi itself work without issues using the same settings? Am I right to suspect that NiFi Registry does LDAP syncronisation in a different way?
2. Is it possible to have the syncronisation start from the defined group(s) and only fetch it's members? There are only about five users in the relevant group, but currently the entire AD gets scanned over and over again.
3. May I specify two OUs for the 'User Search Base'? This would be another option to creatly limit the amount of users being scanned.
4. I couldn't find further details regarding the 'Page Size' setting. Are there any recommendations regarding this configuration parameter?

It's not a real "show stopper" while we simply have to refresh the UI in order to make it working again. Just trying to understand how to optimize the LDAP syncronisation.

Thanks, Chris

[#5]

I do not think the long sync time period is caused by the group search. I think it is caused by the user search. You are correct that it looks like reducing the number of users to search to would require restructuring the AD. I understand that is not always possible / desire-able.

To answer your questions:
- Are you sure the NiFi and NiFi Registry settings are identical? What version of NiFi and NiFi Registry are you using? The LDAP sync procedure should be the same, so I would be surprised if there were very different behaviors.

- No, it is not possible. The algorithm basically pulls all users that match the user search settings then pulls all groups matching the group search settings, then iterates over the users/groups it has synced in order to build the mapping of users<->groups. Changing this sync algorithm would require a re-write of the code or implementing a custom LdapUserGroupProvider.

- It is not possible to specify two OUs for the User Search Base. You could fork the code and implement a modified/custom LdapUserGroupProvider that takes an array of search bases and does multiple searches (one for each base), but that capability is not something currently supported.

- For large directories, I've seen people have good luck specifying a page sizes of 100-500.

Kevin

[#6]

Thanks for the insight, @kdoran

Still trying to limit the amount of users somehow, but apparently its either restructuring the AD or writing a custom LdapUserGroupProvider. Both soloutions are out of scope unfortunately.

Yes, the settings are identical in NiFi and NiFi Registry. Actually I copied the 'authorizers.xml' from our NiFi instance and certainly didn't expect to run into the described issue. I compared both files and only the classes differ.

My conclusion was that something has to work differently with the registry's LDAP sync. I thought of upper-/lowercase, escape characters or other syntax problems, but the docs didn't provide any clues.

We used the official docker images for our deployment, so we are on NiFi 1.8.0 and NiFi Registry 0.3.0 right now.

Will look into the page size property as well. Thanks.

Chris

[#7]

@kdoran

Thank you very much for your suggestions, but 'User Search Base' and 'Group Search Base' are already set accordingly.

• We do have a specific OU containing the NiFi relevant groups. This is were the 'Group Search Base' points to and it only contains two groups.
• It is necessary to define the base OU for users, because there is no specific OU for NiFi relevant users. Therefore the 'User Search Filter' is set to filter only members of a single group (CN=main-nifi-user-group).

Unfortunately I don't see a way to further narrow down the syncing process without changing the structure of the AD itself. I followed your advice and increased the 'Sync Interval' to 120 minutes. Hopefully we won't face inaccessible UI elements too often with this setting.

However, I still got some questions...

1. Why does NiFi itself work without issues using the same settings? Am I right to suspect that NiFi Registry does LDAP syncronisation in a different way?
2. Is it possible to have the syncronisation start from the defined group(s) and only fetch it's members? There are only about five users in the relevant group, but currently the entire AD gets scanned over and over again.
3. May I specify two OUs for the 'User Search Base'? This would be another option to creatly limit the amount of users being scanned.
4. I couldn't find further details regarding the 'Page Size' setting. Are there any recommendations regarding this configuration parameter?

It's not a real "show stopper" while we simply have to refresh the UI in order to make it working again. Just trying to understand how to optimize the LDAP syncronisation.

Thanks, Chris