In order to utilize the directory server's nested groups information across hadoop, nested groups information must be retrieved for hadoop group mapping as well as for Ranger policy authorization. Retrieving nested group information for Ranger is explained here. This article covers the configuration required for retrieving nested group information for hadoop LdapGroupMapping. Nested group membership in LdapGroupMapping is introduced as part of HADOOP-12291 and is available in HDP 2.6. Let's take the same directory server structure example that is used for Ranger. Sample Active directory structure with Nested groups Usecase: Admin wants "Marketing Group" and "AMER Marketing Group" to be represented as hdfs groups for users "Adam Will", "John Doe", and "Mary Sam" In the above example, user "John Doe" is a member of "US Marketing Group". But the directory server structure also contains multiple nested group levels like - “US Marketing Group” is a member of “AMER Marketing Group” which again is a member of “Marketing Group”. With out nested group membership support on hadoop, "hdfs groups" for user "John Doe" returns only the immediate group "US Marketing Group". In order for hdfs groups to retrieve parent groups like "AMER Marketing Group" and "Marketing Group", then the LdapGroupMapping must be configured with nested group membership information. Hadoop LdapGroupMapping configuration ... View more

Many deployments in hadoop use Centrify LDAP proxy to integrate Active Directory users with linux accounts. Centrify ldapproxy uses posix properties like uid, gidNumber, uidNumber, memberUid, etc... and are made available for ldap search attributes to the clients like Apache Ranger. Ranger interacts with Centrify ldapproxy for configuring hadoop authorization policies as well as for authentication to ranger UI. In order to sync users and groups from Centrify, ranger performs ldap search against Centrify using all the configured LDAP attributes. Some of the common issues noticed during this integration can be: Issue #1. Users are unable to login to Ranger UI but were successfully sync'd to ranger: For AD/LDAP authentication, Ranger uses the complete DN of the login user for performing an LDAP search. Ranger also uses group search base and group search filter for group membership for obtaining user role information. Few configuration parameters to verify on ranger are: ranger.ldap.user.dnpattern - This value should match the exact DN pattern of the login user Group Search Base - This value should be the search base of the groups which contains the login users as members Group Search Filter - This value can be a simple filter like "cn=*" Let's look at one small example: ========================================= Ldapsearch with Centrify for one of the user contains following information: dn: cn=alice bob,ou=Users,ou=us,ou=Contractor,dc=apache,dc=org description: c001038@apache.org email: abob@apache.org gecos: Alice Bob gidNumber: 422222198 homeDirectory: /home/c001038 loginShell: /bin/bash uid: c001038 uidNumber: 42222045 userPassword:: WWIEphZ2FkZWVzYW4= accountExpires: 131470704000000000 badPasswordTime: 131140254848671173 badPwdCount: 0 cn: Alice Bob codePage: 0 countryCode: 0 displayName: Alice Bob distinguishedName: CN=Alice Bob,OU=Users,OU=us,OU=C ontractor,DC=apache,DC=org givenName: Alice instanceType: 4 lastLogoff: 0 lastLogon: 131140358889710142 logonCount: 185 mail: Abob@hortonworks.com manager: CN=Joe Dave,OU=Corporate,OU=Users,OU=us,DC=apache,DC=org memberOf: CN=hdp_admin,OU=Hadoop,OU=Applications,OU=Groups,DC=apache,DC=org memberOf: cn=hdp_group1,ou=Hadoop,ou=Groups,dc=apache,dc=org name: Alice Bob objectClass: top objectClass: posixaccount objectGUID:: W6rekfmZXECSiIgFeOTnPw== objectSid:: AQUAAAAAAAUVAAAAYKiAHXLIYdQjI9RcxRQCAA== primaryGroupID: 500 pwdLastSet: 131123031899439530 sAMAccountName: c001038 sAMAccountType: 805306300 sn: Bob uSNChanged: 153583075 uSNCreated: 151863255 userAccountControl: 510 userPrincipalName: c001038@apache.org whenChanged: 20160716210250.0Z whenCreated: 20160705200144.0Z As you can observe the complete DN of the user is configured as “dn: cn=alice bob,ou=Users,ou=us,ou=Contractor,dc=apache,dc=org”. But say the user is using uid (c001038) for logging in to ranger admin and the “ranger.ldap.user.dnpattern” is configured as “uid={0},ou=Users,ou=us,ou=Contractor,dc=apache,dc=org”. In this case the ranger login fails because of the mismatch DN pattern. In order to have a successful login with uid, then Centrify config has to be changed to match the DN to be “uid=c001038,ou=Users,ou=us,ou=Contractor,dc=apache,dc=org”. Otherwise, the user can use cn (alice bob) as the login user and “ranger.ldap.user.dnpattern” can be configured as “cn={0},ou=Users,ou=us,ou=Contractor,dc=apache,dc=org” Issue #2. Groups not sync'd to Ranger: In order to provide flexibility and support different deployment scenarios, ranger provides multiple ways to configure group sync. 1. Sync groups from User’s Group name attribute (like “memberof” or “ismemberof”) - One of the use case here is to “Get all the users from the configured user search base and user search filter and get all the groups that the users belong to” In this case “Group User Map Sync” is set to “true” and “Enable Group Sync” is set to false. The only way ranger gets the group information is by using the User’s Group name attribute (which is generally “memberof” or “ismemberof” attribute of the user). Since Centrify ldapproxy uses posix attributes, most of the times “memberof” or “ismemberof” attribute is not available for the user. In this case groups will not be sync’d to Ranger. There are two ways to solve this issue - a. Without any changes to Centrify, Ranger can be configured with “Enable Group Sync” option (#2 below). b. Modify Centrify to populate “memberof” attribute for the user as in example above (memberOf:cn=hdp_admin,ou=Hadoop,ou=Applications, ou=Groups,dc=apache,dc=org memberOf: cn=hdp_group1,ou=Hadoop,ou=Groups,dc=apache,dc=org) 2. Sync groups by computing group membership of the users - One of the use case here is to “Get all the users from the configured user search base and user search filter and get only particular groups that these users belong to”. This also targets the case where LDAP server doesn’t support “memberof” attribute for the user. In this case “Group User Map Sync” should be set to “true” and “Enable Group Sync” is also should be set to true. In this case ranger first syncs all the users based on the “User Info” configuration. Then for each user ranger computes the group memberships based on the “Group Member Attribute” (like member, memberUID, etc…). Some of the common scenarios where the groups fail to sync to ranger are: a. When “Group User Map Sync” or “Enable Group Sync” not set to true b. When Group search base or group search filter are misconfigured and no matching groups are found. c. When “username” that is configured for User search is not matching the value for configured Group Member Attribute. For this let’s take an example as below: ==================================== Let’s say “User name Attribute” is configured as “uid” and “Group Member Attribute” is configured as “member” in Ranger. Following is the ldapsearch output for a group from Centrify: dn: hdp_group1,ou=Hadoop,ou=Groups,dc=apache,dc=org cn: hdp_group1 gidNumber: 478287198 memberUid: bdave memberUid: c001038 memberUid: c001140 memberUid: djoe groupType: -2122223646 member: <GUID=0b01714af9d97047aa0d55d813a20385>;<SID=0105000000000005150000006 0a8801d72c861d42323d45ce7170200>;CN=Dave Joe,OU=Users,OU=us,OU=Contractor,DC=apache,DC=org member: <GUID=3fef7ee4aa1fd84092fce8d25cfed798>;<SID=0105000000000005150000006 0a8801d72c861d42323d45ca5170200>;CN=bob dave,OU=Users,OU=us,DC=apache,DC=org member: <GUID=5baade91f9995c409288880578e4e73f>;<SID=0105000000000005150000006 0a8801d72c861d42323d45cc5140200>;CN=Alice Bob,OU=Users,OU=us,OU=C ontractor,DC=apache,DC=org objectClass: top objectClass: posixgroup sAMAccountName: hdp_grp1 uSNChanged: 152222220 Since “User name Attribute” in ranger is configured to uid, the users are sync’d to ranger using this attribute (like “c001038”). And for computing group memberships for this user, ranger uses “Group Member Attribute” which is configured as “member”. Ranger performs ldap search to get all the group configured in the group search base that contains the “member” as “uid=c001038”. But as you can see the “member” attribute returned from Centrify doesn’t contain info with uid of the user. Hence the no groups are sync’d to ranger. This issue can be fixed by configuring “Group Member Attribute” in ranger as “memberuid” so that it matches with the value of uid. 3. Sync groups from the configured search base and group search filter - One of the use case here is to “Get all the groups from the configured group search base and group search filter”. This also targets the use case to sync groups with no members. In this case“Group User Map Sync”, “Enable Group Sync” and “Enable Group Search First” should be set to true. The only reason that the groups are not sync’d to ranger because of the misconfiguration of Group search base or group search filter. Issue #3: Issues with Syncing ranger users and groups with centrify AD: Ranger retrieves only the values of the attributes configured in ranger. Hence for syncing users and/or groups ranger performs ldap search against the server (like Centrify) with all these attributes. When users and/or groups not sync’d to ranger, perform ldap search using ldapsearch utility with any particular user and/or group and with the configured attributes. Sometimes a particular attribute (like “memberof”, “member”, etc…) are not configured in Centrify and hence the user sync fails. In this case following error can be found in ranger usersync logs (/var/log/ranger/usersync/usersync.log): javax.naming.directory.NoSuchAttributeException: [LDAP: error code 16 - cdcLdapSearch :No such attribute (cdcRC=8), errSystem=Ldap, errCode=16, errString=No such attribute]; In general since Centrify uses posix attributes, “member” attribute might not be configured for a group. Instead the users are mapped to groups using “memberuid” attribute. In order to solve this issue, verify the “Group Member Attribute” is configured accordingly (memberuid in this case) in Ranger. ... View more