Ranger cannot sync Freeipa's users with groups

[#1]

Hi everyone,

We installed ranger user-sync and able to sync all external users from FreeIpa.But this user-sync can only sync users with groups thoses dont exist in ranger.If a user exists in ranger, this user-sync cannot sync its groups.

We tried to call Ranger Api 

/service/xusers/secure/users/%s

to add user's group.But  when we restart user-sync manually, the user's group added before is gone.

How can we solve this problem ?

[#2]

@pminovic2, Welcome to our community! To help you get the best possible answer, I have tagged in our Ranger experts @xsmehul, @vamsi_redd @Kartik_Agarwal  who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#3]

Hi @pminovic2 ,

Could you please check in /var/log/ranger/usersync/usersync.log what exactly you are getting the error while syncing the group. User group syncing actually works with the user search filter and group search filter configs you provide. Please provide the screenshot of the issue which you are referring and also, make sure the configs are correct as per below document

https://docs.cloudera.com/cdp-private-cloud-base/7.1.3/security-ranger-authentication-unix-ldap-ad/t...

[#4]

Hi @vamsi_redd ,

These are the main usersync.log lines,but there are no error logs:

2023-05-09 10:42:02,759 INFO org.apache.ranger.authentication.UnixAuthenticationService: Starting User Sync Service!
2023-05-09 10:42:02,759 INFO org.apache.ranger.authentication.UnixAuthenticationService: Start : startUnixUserGroupSyncProcess 
2023-05-09 10:42:02,760 INFO org.apache.ranger.authentication.UnixAuthenticationService: UnixUserSyncThread started
2023-05-09 10:42:02,760 INFO org.apache.ranger.authentication.UnixAuthenticationService: creating UserSyncMetricsProducer thread with default metrics location : /var/log/ranger/usersync
2023-05-09 10:42:02,798 INFO org.apache.ranger.authentication.UnixAuthenticationService: UserSyncMetricsProducer started
2023-05-09 10:42:02,801 INFO org.apache.ranger.unixusersync.config.UserGroupSyncConfig: Sleep Time Between Cycle can not be lower than [3600000] millisec. resetting to min value.
2023-05-09 10:42:02,802 INFO org.apache.ranger.usergroupsync.UserSyncMetricsProducer: user sync metrics frequency :  60000 and metrics file : /var/log/ranger/metrics-usersync/metrics.json
2023-05-09 10:42:02,811 INFO org.apache.ranger.usergroupsync.AbstractMapper: Initializing for ranger.usersync.mapping.username.regex
2023-05-09 10:42:02,811 INFO org.apache.ranger.usergroupsync.AbstractMapper: Initializing for ranger.usersync.mapping.groupname.regex
2023-05-09 10:42:02,812 INFO org.apache.ranger.usergroupsync.UserGroupSync: initializing sink: org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder
2023-05-09 10:42:03,429 INFO org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder: Using principal = rangerusersync/dipper-dev-dp-cdp06.xxxx.com@xxxx.COM and keytab = /var/run/cloudera-scm-agent/process/9798-ranger-RANGER_USERSYNC/ranger.keytab
2023-05-09 10:42:03,700 INFO org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder: valid cookie saved 
2023-05-09 10:42:03,730 INFO org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder: PolicyMgrUserGroupBuilder.buildGroupList(): No. of groups retrieved from ranger admin 389
2023-05-09 10:42:05,512 INFO org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder: PolicyMgrUserGroupBuilder.buildUserList(): No. of users retrieved from ranger admin = 484
2023-05-09 10:42:05,523 INFO org.apache.ranger.usergroupsync.UserGroupSync: initializing source: org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder
2023-05-09 10:42:05,523 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: LdapUserGroupBuilder initialization started
2023-05-09 10:42:05,619 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: LdapUserGroupBuilder initialization completed with --  ldapUrl: ldap://zj1-dipper10-cdh-ipamaster.xxxx.com:389,  ldapBindDn: uid=admin,cn=users,cn=accounts,dc=xxxx,dc=com,  ldapBindPassword: ***** ,  ldapAuthenticationMechanism: simple,  searchBase: null,  userSearchBase: [cn=users,cn=accounts,dc=xxxx,dc=com],  userSearchScope: 2,  userObjectClass: person,  userSearchFilter: null,  extendedUserSearchFilter: null,  userNameAttribute: uid,  userSearchAttributes: [uid, uSNChanged, memberof, ismemberof, modifytimestamp, objectid, userurincipaluame],  userGroupNameAttributeSet: [memberof, ismemberof],  otherUserAttributes: [userurincipaluame],  pagedResultsEnabled: true,  pagedResultsSize: 500,  groupSearchEnabled: true,  groupSearchBase: [cn=groups,cn=accounts,dc=xxxx,dc=com],  groupSearchScope: 2,  groupObjectClass: groupofnames,  groupSearchFilter: null,  extendedGroupSearchFilter: (&null(|(member={0})(member={1}))),  extendedAllGroupsSearchFilter: null,  groupMemberAttributeName: member,  groupNameAttribute: cn, groupSearchAttributes: [uSNChanged, displayname, member, cn, modifytimestamp, objectid], groupSearchFirstEnabled: true, userSearchEnabled: true,  ldapReferral: ignore
2023-05-09 10:42:05,620 INFO org.apache.ranger.usergroupsync.UserGroupSync: Begin: initial load of user/group from source==>sink
2023-05-09 10:42:05,620 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: LdapUserGroupBuilder updateSink started
2023-05-09 10:42:05,631 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: extendedAllGroupsSearchFilter = (&(objectclass=groupofnames)(|(uSNChanged>=0)(modifyTimestamp>=19700101080000Z)))
...
2023-05-09 10:42:05,706 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: timeStampVal = 20230509024154Zand currentDeltaSyncTime = 1683571314000
2023-05-09 10:42:05,707 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: No. of members in the group t_person = 60
...
2023-05-09 10:42:05,708 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: LdapUserGroupBuilder.getGroups() completed with group count: 371
2023-05-09 10:42:05,709 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: Performing user search to retrieve users from AD/LDAP
2023-05-09 10:42:05,712 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: extendedUserSearchFilter = (&(objectclass=person)(|(uSNChanged>=0)(modifyTimestamp>=19700101080000Z)))
...
2023-05-09 10:42:05,945 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: timeStampVal = 20230506020930Zand currentDeltaSyncTime = 1683310170000
2023-05-09 10:42:05,945 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: Updating user count: 444, userName: test20230506
2023-05-09 10:42:05,945 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: timeStampVal = 20230506022758Zand currentDeltaSyncTime = 1683311278000
2023-05-09 10:42:05,945 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: Updating user count: 445, userName: test2022305061027
2023-05-09 10:42:05,945 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: timeStampVal = 20230509024154Zand currentDeltaSyncTime = 1683571314000
2023-05-09 10:42:05,945 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: Updating user count: 446, userName: test20230509
2023-05-09 10:42:05,945 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: LdapUserGroupBuilder.getUsers() completed with user count: 446
2023-05-09 10:42:06,106 INFO org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder: ret = 1 No. of users uploaded to ranger admin= 1
2023-05-09 10:42:06,151 INFO org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder: ret = 2 No. of group memberships uploaded to ranger admin= 2
2023-05-09 10:42:06,152 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: deltaSyncUserTime = 0 and highestdeltaSyncUserTime = 1683571314000
2023-05-09 10:42:06,152 INFO org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder: deltaSyncGroupTime = 0 and highestdeltaSyncGroupTime = 1683571314000
2023-05-09 10:42:06,175 INFO org.apache.ranger.usergroupsync.UserGroupSync: End: initial load of user/group from source==>sink
2023-05-09 10:42:06,175 INFO org.apache.ranger.usergroupsync.UserGroupSync: Done initializing user/group source and sink
2023-05-09 10:42:07,798 INFO org.apache.ranger.authentication.UnixAuthenticationService: Enabling Unix Auth Service!
2023-05-09 10:42:07,957 INFO org.apache.ranger.authentication.UnixAuthenticationService: Disabling Protocol: [TLSv1.3]
2023-05-09 10:42:07,957 INFO org.apache.ranger.authentication.UnixAuthenticationService: Enabling Protocol: [TLSv1.2]
2023-05-09 10:42:07,957 INFO org.apache.ranger.authentication.UnixAuthenticationService: Disabling Protocol: [SSLv2Hello]

eg: the user test20230506 in FreeIpa has the group t_person,and test20230506 is an external user in ranger.The ranger user-sync cannot give the group t_person to test20230506. And After I tried to call Ranger Api 

/service/xusers/secure/users/%s

to add the group t_person to test20230506, the group t_person is added.But when user-sync service retstarts the user test20230506's groups are gone.The Ranger's version is 2.1

[#5]

Can you please provide the ldapsearch output for both test20230506 and t_person

[#6]

1. command: ldapsearch -h localhost -p 389 -D "uid=admin,cn=users,cn=accounts,dc=xxxx,dc=com" -b "cn=users,cn=accounts,dc=xxxx,dc=com" "(&(objectclass=person)(|(uSNChanged>=0)(modifyTimestamp>=19700101080000Z)))" -W

output for both test20230506：

# test20230506, users, accounts, xxxx.com
dn: uid=test20230506,cn=users,cn=accounts,dc=xxxx,dc=com
memberOf: cn=ipausers,cn=groups,cn=accounts,dc=xxxx,dc=com
memberOf: ipaUniqueID=267b9f7e-15f6-11ec-92c2-005056a46ab7,cn=sudorules,cn=sud
 o,dc=xxxx,dc=com
memberOf: cn=t_person,cn=groups,cn=accounts,dc=xxxx,dc=com
ipaNTSecurityIdentifier: S-1-5-21-1820330004-495089621-1560112186-2228
mepManagedEntry: cn=test20230506,cn=groups,cn=accounts,dc=xxxx,dc=com
krbExtraData:: ******
krbLastPwdChange: 20230506020929Z
krbPasswordExpiration: 20230506020929Z
displayName: test20230506 test20230506
cn: test20230506 test20230506
krbCanonicalName: test20230506@xxxx.COM
objectClass: top
objectClass: person
objectClass: organizationalperson
objectClass: inetorgperson
objectClass: inetuser
objectClass: posixaccount
objectClass: krbprincipalaux
objectClass: krbticketpolicyaux
objectClass: ipaobject
objectClass: ipasshuser
objectClass: ipaSshGroupOfPubKeys
objectClass: mepOriginEntry
objectClass: ipantuserattrs
loginShell: /bin/sh
initials: tt
gidNumber: 523401156
gecos: test20230506 test20230506
sn: test20230506
homeDirectory: /home/test20230506
uid: test20230506
mail: test20230506@xxxx.com
krbPrincipalName: test20230506@xxxx.COM
givenName: test20230506
ipaUniqueID: 08fd4698-ebb3-11ed-bc87-005056a46ab7
uidNumber: 523401228

2.command: ldapsearch -h localhost -p 389 -D "uid=admin,cn=users,cn=accounts,dc=xxxx,dc=com" -b "cn=groups,cn=accounts,dc=xxxx,dc=com" "(&(objectclass=groupofnames)(|(uSNChanged>=0)(modifyTimestamp>=19700101080000Z)))" -W

output for  t_person:

# t_person, groups, accounts, xxxx.com
dn: cn=t_person,cn=groups,cn=accounts,dc=xxxx,dc=com
...
member: uid=test20230506,cn=users,cn=accounts,dc=xxxx,dc=com
member: uid=test2022305061027,cn=users,cn=accounts,dc=xxxx,dc=com
member: uid=test20230509,cn=users,cn=accounts,dc=xxxx,dc=com
ipaNTSecurityIdentifier: S-1-5-21-1820330004-495089621-1560112186-2156
objectClass: top
objectClass: groupofnames
objectClass: nestedgroup
objectClass: ipausergroup
objectClass: ipaobject
objectClass: posixgroup
objectClass: ipantgroupattrs
cn: t_person
ipaUniqueID: 37662654-cc4a-11ed-b308-005056a46ab7
gidNumber: 523401156

[#7]

2.command：ldapsearch -h localhost -p 389 -D "uid=admin,cn=users,cn=accounts,dc=xxxx,dc=com" -b "cn=groups,cn=accounts,dc=xxxx,dc=com" "(&(objectclass=groupofnames)(|(uSNChanged>=0)(modifyTimestamp>=19700101080000Z)))" -W

output for t_person

# t_person, groups, accounts, xxxx.com
dn: cn=t_person,cn=groups,cn=accounts,dc=xxxx,dc=com
...
member: uid=test20230506,cn=users,cn=accounts,dc=xxxx,dc=com
member: uid=test2022305061027,cn=users,cn=accounts,dc=xxxx,dc=com
member: uid=test20230509,cn=users,cn=accounts,dc=xxxx,dc=com
ipaNTSecurityIdentifier: S-1-5-21-1820330004-495089621-1560112186-2156
objectClass: top
objectClass: groupofnames
objectClass: nestedgroup
objectClass: ipausergroup
objectClass: ipaobject
objectClass: posixgroup
objectClass: ipantgroupattrs
cn: t_person
ipaUniqueID: 37662654-cc4a-11ed-b308-005056a46ab7
gidNumber: 523401156

[#8]

Thanks for the output, similarly please provide for user test20230506 as well.  And are the other users like test2022305061027, test20230509 syncing correctly?

[#9]

ldapsearch -h localhost -p 389 -D "uid=admin,cn=users,cn=accounts,dc=xxxx,dc=com" -b "cn=users,cn=accounts,dc=xxxx,dc=com" "(&(objectclass=person)(|(uSNChanged>=0)(modifyTimestamp>=19700101080000Z)))" -W

output for  test20230506

# test20230506, users, accounts, xxxx.com
dn: uid=test20230506,cn=users,cn=accounts,dc=xxxx,dc=com
memberOf: cn=ipausers,cn=groups,cn=accounts,dc=xxxx,dc=com
memberOf: ipaUniqueID=267b9f7e-15f6-11ec-92c2-005056a46ab7,cn=sudorules,cn=sud
 o,dc=xxxx,dc=com
memberOf: cn=t_person,cn=groups,cn=accounts,dc=xxxx,dc=com
ipaNTSecurityIdentifier: S-1-5-21-1820330004-495089621-1560112186-2228
mepManagedEntry: cn=test20230506,cn=groups,cn=accounts,dc=xxxx,dc=com
krbExtraData:: ******
krbLastPwdChange: 20230506020929Z
krbPasswordExpiration: 20230506020929Z
displayName: test20230506 test20230506
cn: test20230506 test20230506
krbCanonicalName: test20230506@xxxx.COM
objectClass: top
objectClass: person
objectClass: organizationalperson
objectClass: inetorgperson
objectClass: inetuser
objectClass: posixaccount
objectClass: krbprincipalaux
objectClass: krbticketpolicyaux
objectClass: ipaobject
objectClass: ipasshuser
objectClass: ipaSshGroupOfPubKeys
objectClass: mepOriginEntry
objectClass: ipantuserattrs
loginShell: /bin/sh
initials: tt
gidNumber: 523401156
gecos: test20230506 test20230506
sn: test20230506
homeDirectory: /home/test20230506
uid: test20230506
mail: test20230506@xxxx.com
krbPrincipalName: test20230506@xxxx.COM
givenName: test20230506
ipaUniqueID: 08fd4698-ebb3-11ed-bc87-005056a46ab7
uidNumber: 523401228

[#10]

# test20230506, users, accounts, xxxx.com
dn: uid=test20230506,cn=users,cn=accounts,dc=xxxx,dc=com
memberOf: cn=ipausers,cn=groups,cn=accounts,dc=xxxx,dc=com
memberOf: ipaUniqueID=267b9f7e-15f6-11ec-92c2-005056a46ab7,cn=sudorules,cn=sud
o,dc=xxxx,dc=com
memberOf: cn=t_person,cn=groups,cn=accounts,dc=xxxx,dc=com
ipaNTSecurityIdentifier: S-1-5-21-1820330004-495089621-1560112186-2228
mepManagedEntry: cn=test20230506,cn=groups,cn=accounts,dc=xxxx,dc=com
krbExtraData:: ******
krbLastPwdChange: 20230506020929Z
krbPasswordExpiration: 20230506020929Z
displayName: test20230506 test20230506
cn: test20230506 test20230506
krbCanonicalName: test20230506@xxxx.COM
objectClass: top
objectClass: person
objectClass: organizationalperson
objectClass: inetorgperson
objectClass: inetuser
objectClass: posixaccount
objectClass: krbprincipalaux
objectClass: krbticketpolicyaux
objectClass: ipaobject
objectClass: ipasshuser
objectClass: ipaSshGroupOfPubKeys
objectClass: mepOriginEntry
objectClass: ipantuserattrs
loginShell: /bin/sh
initials: tt
gidNumber: 523401156
gecos: test20230506 test20230506
sn: test20230506
homeDirectory: /home/test20230506
uid: test20230506
mail: test20230506@xxxx.com
krbPrincipalName: test20230506@xxxx.COM
givenName: test20230506
ipaUniqueID: 08fd4698-ebb3-11ed-bc87-005056a46ab7
uidNumber: 523401228