Configuring Ranger Usersync with AD/LDAP is highly dependent on the customer environment. This requires understanding of the customer environment as well as the specific requirements for syncing users and groups. In order to get some background on User management in Hadoop, please refer to this Dataworks Summit Presentation. Corresponding slides are available at: Slides In this article we will take a common use case and see how Ranger Usersync can be configured in that scenario. Ranger Usersync supports various configuration options with AD/LDAP and details are available here Use case: Sync all the users who are members of a specific group(s) This use case can be handled in multiple ways and is also based on the AD/LDAP attributes available in the server. In case of Active Directory, users contain information of the group(s) they belong to using the “memberof” attribute. Similarly groups contain information about the user(s) that belong to them using the “member” attribute. Whereas in default installation of openLdap server doesn’t contain “memberof” attribute for the user. So the only way to retrieve user that belong to a group is by using “member” attribute of the group. Let’s take an example in Active directory where we want to sync all the users that belong to groups - “hdp_testing”, “hdp_admin”, or “dev_ops” ------------------------------------------------------------------------------------------ Sample ldapsearch command to search a particular group (hdp_testing) and determine what attributes are available for the group: ldapsearch -x -LLL -h 10.10.10.10:389 -D 'cn=administrator,CN=Users,dc=hortonworks,dc=com' -W -b 'ou=Hadoop Groups,dc=hortonworks,dc=com' 'cn=hdp_testing’ Enter LDAP Password: dn: CN=hdp_testing,ou=Hadoop Groups,dc=hortonworks,dc=com objectClass: top objectClass: group cn: hdp_testing member: CN=test11 test,ou=Hadoop Users,dc=hortonworks,dc=com member: CN=user12 user,ou=Hadoop Users,dc=hortonworks,dc=com member: CN=user8 user,ou=Hadoop Users,dc=hortonworks,dc=com member: CN=user7 user,ou=Hadoop Users,dc=hortonworks,dc=com member: CN=user6 user,ou=Hadoop Users,dc=hortonworks,dc=com member: CN=user1 user,ou=Hadoop Users,dc=hortonworks,dc=com distinguishedName: CN=hdp_testing,ou=Hadoop Groups,dc=hortonworks,dc=com instanceType: 4 uSNChanged: 1312771 name: group5 sAMAccountName: hdp_testing sAMAccountType: 222431111 Above output shows all the available attributes for hdp_testing. Note:- Highlighted are the attributes that are interested for usersync configuration. In this case hdp_testing has multiple members (highlighted “member” attribute) like member: CN=test11 test,ou=Hadoop Users,dc=hortonworks,dc=com member: CN=user12 user,ou=Hadoop Users,dc=hortonworks,dc=com member: CN=user8 user,ou=Hadoop Users,dc=hortonworks,dc=com member: CN=user7 user,ou=Hadoop Users,dc=hortonworks,dc=com etc… Sample ldapsearch command to search a particular user (user8) and determine what attributes are available for the user: ldapsearch -x -LLL -h 10.10.10.10:389 -D 'cn=administrator,CN=Users,dc=hortonworks,dc=com' -W -b 'ou=Hadoop Users,dc=hortonworks,dc=com' 'samaccountname=user8' Enter LDAP Password: dn: CN=user8 user,ou=Hadoop Users,dc=hortonworks,dc=com objectClass: top objectClass: person objectClass: organizationalPerson objectClass: user cn: user8 user sn: user givenName: user8 distinguishedName: CN=user8 user,ou=Hadoop Users,dc=hortonworks,dc=com instanceType: 4 memberOf: CN=hdp_testing,ou=Hadoop Groups,dc=hortonworks,dc=com memberOf: CN=dev_ops,ou=Hadoop Groups,dc=hortonworks,dc=com memberOf: CN=test_groups,ou=Hadoop Groups,dc=hortonworks,dc=com memberOf: CN=security_groups,ou=groups,dc=hortonworks,dc=com uSNChanged: 973651 userAccountControl: 512 badPwdCount: 0 codePage: 0 countryCode: 0 badPasswordTime: 0 lastLogoff: 0 lastLogon: 0 pwdLastSet: 0 primaryGroupID: 513 logonCount: 0 sAMAccountName: user8 Above output shows all the available attributes for a user. Note:- Highlighted are the attributes that are interested for usersync configuration. In this case user8 is a “memberof” 4 groups - hdp_testing, dev_ops, test_groups, and security_groups As you can see, users from hdp_testing, dev_ops, or hdp_admins can be sync’d to ranger by performing a user search using the “memberof” attribute in the user search filter -- User based search or a group search using the “member” attribute -- Group based search Ranger Usersync configuration contains three sets of configuration: Common Configs: Ldap Url and bind credentials User Configs: Attributes related to user like sAMAccountName, OU(s) of the users, user search filter, memberof, etc… Group Configs: Attributes related to group like sAMAccountName, OU(s) of the groups, group search filter, member, etc… Following is the screenshot of the Common Configs properties configured for the above example: Please note the following: From HDP 2.6 onwards Ranger Usersync supports “Incremental Sync” and is enabled by default. For clusters that are upgraded from older version to 2.6, “Incremental Sync” is disabled. When “Incremental sync” is enabled, “Enable Group Sync” is set to “true” by default and the properties under “Group Configs” is mandatory. Configuring Ranger Usersync for User based search: ============================================ Here Users are searched based on the attributes available from the user attribute and group information is available as part of the “memberof”. In this case two main parts that need to be configured in Ranger Usersync configuration are the properties under - “Common Configs” and “User Configs”. Optionally properties under “Group Configs” can be configured in cases where customers want to limit or filter the groups that these users belong to. As you can see in the above example, user8 belongs to 4 groups - hdp_testing, dev_ops, test_groups, and security_groups. But if we are interested only in groups hdp_testing and dev_ops, then “Group Configs” properties can be set accordingly. Following are the screenshots of User Configs and Group Configs properties configured for the above example with User based search: User Configs: ------------------------------- Group Configs: ------------------------------- Configuring Ranger Usersync for Group based search: ============================================ In this case Ranger Usersync performs group search first based on the group configuration. Here users are searched based on the attributes available from the group attributes and user information is available as part of the “member”. In this case two main parts that need to be configured in Ranger Usersync configuration are the properties under - “Common Configs” and “Group Configs”. Optionally properties under “User Configs” can be configured in cases where customers want to limit or filter the users that belong to these groups. Following are the screenshots of Group Configs, and User Configs properties configured for the above example with Group based search: Group Configs: ------------------------------- Note:- “Enable Group First Search” must be set to “true” in order to perform Group based search. User Configs: ------------------------------- Please note the following: “Enable User Search” is set to false by default. When “Enable User Search” is set to “false” none of the properties under “User Configs” will be used by Ranger Usersync Users are sync’d using group’s member attribute which in many cases contains CN (Firstname Lastname) of the user. In order to sync users using sAMAccountName, then “Enable User Search” must be set to “true” and “Username Attribute” must be set to “sAMAccountName”. Note:- With “Enable User Search” set to true, it is still required to set all other user configs appropriately like the user search base and user object class. User search filter can be set to “sAMAccountName=*” ... View more

Many deployments in hadoop use Centrify LDAP proxy to integrate Active Directory users with linux accounts. Centrify ldapproxy uses posix properties like uid, gidNumber, uidNumber, memberUid, etc... and are made available for ldap search attributes to the clients like Apache Ranger. Ranger interacts with Centrify ldapproxy for configuring hadoop authorization policies as well as for authentication to ranger UI. In order to sync users and groups from Centrify, ranger performs ldap search against Centrify using all the configured LDAP attributes. Some of the common issues noticed during this integration can be: Issue #1. Users are unable to login to Ranger UI but were successfully sync'd to ranger: For AD/LDAP authentication, Ranger uses the complete DN of the login user for performing an LDAP search. Ranger also uses group search base and group search filter for group membership for obtaining user role information. Few configuration parameters to verify on ranger are: ranger.ldap.user.dnpattern - This value should match the exact DN pattern of the login user Group Search Base - This value should be the search base of the groups which contains the login users as members Group Search Filter - This value can be a simple filter like "cn=*" Let's look at one small example: ========================================= Ldapsearch with Centrify for one of the user contains following information: dn: cn=alice bob,ou=Users,ou=us,ou=Contractor,dc=apache,dc=org description: c001038@apache.org email: abob@apache.org gecos: Alice Bob gidNumber: 422222198 homeDirectory: /home/c001038 loginShell: /bin/bash uid: c001038 uidNumber: 42222045 userPassword:: WWIEphZ2FkZWVzYW4= accountExpires: 131470704000000000 badPasswordTime: 131140254848671173 badPwdCount: 0 cn: Alice Bob codePage: 0 countryCode: 0 displayName: Alice Bob distinguishedName: CN=Alice Bob,OU=Users,OU=us,OU=C ontractor,DC=apache,DC=org givenName: Alice instanceType: 4 lastLogoff: 0 lastLogon: 131140358889710142 logonCount: 185 mail: Abob@hortonworks.com manager: CN=Joe Dave,OU=Corporate,OU=Users,OU=us,DC=apache,DC=org memberOf: CN=hdp_admin,OU=Hadoop,OU=Applications,OU=Groups,DC=apache,DC=org memberOf: cn=hdp_group1,ou=Hadoop,ou=Groups,dc=apache,dc=org name: Alice Bob objectClass: top objectClass: posixaccount objectGUID:: W6rekfmZXECSiIgFeOTnPw== objectSid:: AQUAAAAAAAUVAAAAYKiAHXLIYdQjI9RcxRQCAA== primaryGroupID: 500 pwdLastSet: 131123031899439530 sAMAccountName: c001038 sAMAccountType: 805306300 sn: Bob uSNChanged: 153583075 uSNCreated: 151863255 userAccountControl: 510 userPrincipalName: c001038@apache.org whenChanged: 20160716210250.0Z whenCreated: 20160705200144.0Z As you can observe the complete DN of the user is configured as “dn: cn=alice bob,ou=Users,ou=us,ou=Contractor,dc=apache,dc=org”. But say the user is using uid (c001038) for logging in to ranger admin and the “ranger.ldap.user.dnpattern” is configured as “uid={0},ou=Users,ou=us,ou=Contractor,dc=apache,dc=org”. In this case the ranger login fails because of the mismatch DN pattern. In order to have a successful login with uid, then Centrify config has to be changed to match the DN to be “uid=c001038,ou=Users,ou=us,ou=Contractor,dc=apache,dc=org”. Otherwise, the user can use cn (alice bob) as the login user and “ranger.ldap.user.dnpattern” can be configured as “cn={0},ou=Users,ou=us,ou=Contractor,dc=apache,dc=org” Issue #2. Groups not sync'd to Ranger: In order to provide flexibility and support different deployment scenarios, ranger provides multiple ways to configure group sync. 1. Sync groups from User’s Group name attribute (like “memberof” or “ismemberof”) - One of the use case here is to “Get all the users from the configured user search base and user search filter and get all the groups that the users belong to” In this case “Group User Map Sync” is set to “true” and “Enable Group Sync” is set to false. The only way ranger gets the group information is by using the User’s Group name attribute (which is generally “memberof” or “ismemberof” attribute of the user). Since Centrify ldapproxy uses posix attributes, most of the times “memberof” or “ismemberof” attribute is not available for the user. In this case groups will not be sync’d to Ranger. There are two ways to solve this issue - a. Without any changes to Centrify, Ranger can be configured with “Enable Group Sync” option (#2 below). b. Modify Centrify to populate “memberof” attribute for the user as in example above (memberOf:cn=hdp_admin,ou=Hadoop,ou=Applications, ou=Groups,dc=apache,dc=org memberOf: cn=hdp_group1,ou=Hadoop,ou=Groups,dc=apache,dc=org) 2. Sync groups by computing group membership of the users - One of the use case here is to “Get all the users from the configured user search base and user search filter and get only particular groups that these users belong to”. This also targets the case where LDAP server doesn’t support “memberof” attribute for the user. In this case “Group User Map Sync” should be set to “true” and “Enable Group Sync” is also should be set to true. In this case ranger first syncs all the users based on the “User Info” configuration. Then for each user ranger computes the group memberships based on the “Group Member Attribute” (like member, memberUID, etc…). Some of the common scenarios where the groups fail to sync to ranger are: a. When “Group User Map Sync” or “Enable Group Sync” not set to true b. When Group search base or group search filter are misconfigured and no matching groups are found. c. When “username” that is configured for User search is not matching the value for configured Group Member Attribute. For this let’s take an example as below: ==================================== Let’s say “User name Attribute” is configured as “uid” and “Group Member Attribute” is configured as “member” in Ranger. Following is the ldapsearch output for a group from Centrify: dn: hdp_group1,ou=Hadoop,ou=Groups,dc=apache,dc=org cn: hdp_group1 gidNumber: 478287198 memberUid: bdave memberUid: c001038 memberUid: c001140 memberUid: djoe groupType: -2122223646 member: <GUID=0b01714af9d97047aa0d55d813a20385>;<SID=0105000000000005150000006 0a8801d72c861d42323d45ce7170200>;CN=Dave Joe,OU=Users,OU=us,OU=Contractor,DC=apache,DC=org member: <GUID=3fef7ee4aa1fd84092fce8d25cfed798>;<SID=0105000000000005150000006 0a8801d72c861d42323d45ca5170200>;CN=bob dave,OU=Users,OU=us,DC=apache,DC=org member: <GUID=5baade91f9995c409288880578e4e73f>;<SID=0105000000000005150000006 0a8801d72c861d42323d45cc5140200>;CN=Alice Bob,OU=Users,OU=us,OU=C ontractor,DC=apache,DC=org objectClass: top objectClass: posixgroup sAMAccountName: hdp_grp1 uSNChanged: 152222220 Since “User name Attribute” in ranger is configured to uid, the users are sync’d to ranger using this attribute (like “c001038”). And for computing group memberships for this user, ranger uses “Group Member Attribute” which is configured as “member”. Ranger performs ldap search to get all the group configured in the group search base that contains the “member” as “uid=c001038”. But as you can see the “member” attribute returned from Centrify doesn’t contain info with uid of the user. Hence the no groups are sync’d to ranger. This issue can be fixed by configuring “Group Member Attribute” in ranger as “memberuid” so that it matches with the value of uid. 3. Sync groups from the configured search base and group search filter - One of the use case here is to “Get all the groups from the configured group search base and group search filter”. This also targets the use case to sync groups with no members. In this case“Group User Map Sync”, “Enable Group Sync” and “Enable Group Search First” should be set to true. The only reason that the groups are not sync’d to ranger because of the misconfiguration of Group search base or group search filter. Issue #3: Issues with Syncing ranger users and groups with centrify AD: Ranger retrieves only the values of the attributes configured in ranger. Hence for syncing users and/or groups ranger performs ldap search against the server (like Centrify) with all these attributes. When users and/or groups not sync’d to ranger, perform ldap search using ldapsearch utility with any particular user and/or group and with the configured attributes. Sometimes a particular attribute (like “memberof”, “member”, etc…) are not configured in Centrify and hence the user sync fails. In this case following error can be found in ranger usersync logs (/var/log/ranger/usersync/usersync.log): javax.naming.directory.NoSuchAttributeException: [LDAP: error code 16 - cdcLdapSearch :No such attribute (cdcRC=8), errSystem=Ldap, errCode=16, errString=No such attribute]; In general since Centrify uses posix attributes, “member” attribute might not be configured for a group. Instead the users are mapped to groups using “memberuid” attribute. In order to solve this issue, verify the “Group Member Attribute” is configured accordingly (memberuid in this case) in Ranger. ... View more