In order to utilize the directory server's nested groups information across hadoop, nested groups information must be retrieved for hadoop group mapping as well as for Ranger policy authorization. Retrieving nested group information for Ranger is explained here. This article covers the configuration required for retrieving nested group information for hadoop LdapGroupMapping. Nested group membership in LdapGroupMapping is introduced as part of HADOOP-12291 and is available in HDP 2.6. Let's take the same directory server structure example that is used for Ranger. Sample Active directory structure with Nested groups Usecase: Admin wants "Marketing Group" and "AMER Marketing Group" to be represented as hdfs groups for users "Adam Will", "John Doe", and "Mary Sam" In the above example, user "John Doe" is a member of "US Marketing Group". But the directory server structure also contains multiple nested group levels like - “US Marketing Group” is a member of “AMER Marketing Group” which again is a member of “Marketing Group”. With out nested group membership support on hadoop, "hdfs groups" for user "John Doe" returns only the immediate group "US Marketing Group". In order for hdfs groups to retrieve parent groups like "AMER Marketing Group" and "Marketing Group", then the LdapGroupMapping must be configured with nested group membership information. Hadoop LdapGroupMapping configuration ... View more

Introduction Adding a group as a member to another group is called nesting. If a group (groupA) is a member of another group (groupB), then the users belonging to the member group (groupA) are part of the parent group (groupB) as well. Nesting can be very useful in delegating access through inheritance. Several large enterprises have their groups in LDAP/AD nested within other groups. Security admins want the users in those nested groups to be associated in Ranger so that they are available for policy authoring in Ranger Admin. In HDP 2.6 (RANGER-1735), ranger usersync introduced support of nested group membership representation for policy authoring. Note:- In order to utilize the nested group mapping across hadoop, this feature must be configured for hadoop LdapGroupMapping as well. Configuring hadoop LdapGroupMapping for nested groups is explained here. Sample Active directory structure with Nested groups Usecase: Admin wants to give access to some resources for all the users under “AMER Marketing Group” In the above sample directory structure, all the marketing users are under one OU “Marketing Users”. All these users are members of different groups based on the location like US, Canada, London, etc… For example, user “Adam Will” from “Marketing Users” OU is a member of “Canada Marketing Group”. Also, the above sample directory structure contains multiple nested group levels like - “US Marketing Group” is a member of “AMER Marketing Group” which again is a member of “Marketing Group”. Ranger Usersync configuration Ranger Usersync, by default, computes only the immediate groups for the users. For example, user “Adam Will” is part of “Canada Marketing Group” and only this information is available in ranger without nested group sync configuration. With this information, if an admin wants to provide access to all the users under “AMER Marketing Group”, then all the sub groups - “US Marketing Group” and “Canada Marketing Group” must be added in the ranger policy. In order to simplify the policy configuration at parent level groups, Ranger supports evaluating nested group memberships by configuring “ranger.usersync.ldap.grouphierarchylevels”. If ranger.usersync.ldap.grouphierarchylevels is set to “3”, Ranger Usersync computes the group memberships for user “Adam Will” as “Canada Marketing Group”, “AMER Marketing Group”, “Marketing Group”. This way, admin can configure ranger policy at the parent group level (“AMER Marketing Group”) which will be applied for all the users (Mary Sam, John Doe, and Adam Will) under each sub group (US Marketing Group and Canada Marketing Group). ... View more

This article mainly focuses mainly on what all supported options available in User sync and how they can be used to target specific use cases. For some common use cases examples and how ranger can be configured in those cases, please refer to Configuring Ranger UserSync with AD/LDAP User based search without enable group sync: This configuration can be used in cases where customers want to sync users based on a OU or a user search filter and get all the groups the users belong to. This case is supported only when the user object has the group membership attributes like “memberof” or “ismemberof”. This option is available only in Full sync case (i.e., when Incremental sync is disabled in Ranger User sync configuration) User based search with enable group sync: This configuration can be used in cases where - customers want to sync users based on a OU or a user search filter and want to limit the groups that these user belong to based on the OU or a group search filter. Group membership attributes like “memberof” is not available for the user objects. This is the default option when “Incremental Sync” is enabled in Ranger Usersync configuration This option is available for both “Incremental Sync” or “Full Sync” Group based search without enable user sync (RANGER-869): This option is available for both “Incremental Sync” or “Full Sync” This option cannot be used in cases where customers want to sync the users with “sAMAccountName” instead if “CN” This is the default option when “Group Search First Enable” is set to “true” This configuration can be used in cases where - customers want to sync groups based on a OU or a group search filter and get all the users that belong to those groups. customers want to sync the groups that don’t have any users Group based search with enable user sync (RANGER-869): This configuration can be used in cases where customers want to sync groups based on a OU or a group search filter and limit the users that belong to those groups based on a OU or a user search filter. customers want to sync groups based on a OU or a group search filter and use the “sAMAccountName” for the user instead of “CN” customers want to sync the groups that don’t have any users This option is available for both “Incremental Sync” or “Full Sync” Multiple OUs for User search base and/or Group search base (RANGER-803): This configuration can be used in cases where customers want to sync users and/or groups from multiple OUs Multiple OUs can be specified in the User search base and/or Group search base using “;” separated like ou=OU1,dc=hortonworks,dc=com;ou=OU2,dc=hortonworks,dc=com;ou=OU3,dc=hortonworks,dc=com Multiple OUs are supported for both User based search and Group based search Incremental Sync or Full sync (RANGER-1211): Incremental Sync - Ranger Usersync can be configured to perform full sync only during startup and incremental sync for the subsequent sync cycles. This is the default option for fresh installs of HDP version 2.6 onwards With this option, “Enable Group Sync” is mandatory This option is highly recommended in order to improve usersync performance and decrease the startup time of Ranger. Full sync - Ranger Usersync can be configured to perform full sync from AD/LDAP for every sync cycle. This is the default option for clusters upgraded to 2.6 or higher Username and/or Groupname case conversion: Ranger usersync support three options for case conversion of username and/or groupname - “lower”, “upper”, and “none” “None” is the default option for case conversion This option must be configured to be same as the hdfs users/groups as ranger authorization is case sensitive for username and/or groupname. Ranger usersync properties for case conversion are: ranger.usersync.ldap.username.caseconversion ranger.usersync.ldap.groupname.caseconversion Username and/or Groupname transformation (RANGER-684): Ranger Usersync can be configured to perform username and/or groupname transformations in order to make them POSIX compliant. The value for these properties should be in the following format (similar to Sed format):“s/regex/replacement/g”s – stands for substitution and is mandatory/ - delimiter and is mandatoryRegex – regular expression to matchReplacement – value to replace and is optional. If not specified, the found pattern is removed from the resulting string.g – for replacing all the occurrences. It is optional and if not specified, default to “g” as well. In order to enable this feature, following properties should be added as custom usersync site properties in Ambari: ranger.usersync.mapping.username.handler - value as “org.apache.ranger.usergroupsync.RegEx” ranger.usersync.mapping.groupname.handler - value as “org.apache.ranger.usergroupsync.RegEx” ranger.usersync.mapping.username.regex - value must be in the format defined in section #b above ranger.usersync.mapping.groupname.regex - value must be in the format defined in #b above To configure multiple transformations/mappings: new regex properties can be added as: ranger.usersync.mapping.<attribute name>.<tranformation>.1, ranger.usersync.mapping.<attribute name>.<tranformation>.2, etc… All these transformations will be applied in the order they were configured. LDAP or LDAPS or STARTTLS (RANGER-722): Ranger Usersync support both secure and non-secure communication to AD/LDAP server for performing the sync operation. Ranger usersync can be configure with LDAPS or STARTTLS for secure communication For LDAP/LDAPS, ldap url must be configured accordingly with the corresponding port. For example - For non-secure/LDAP communication, sample ldap url value is “ldap://10.10.10.10:389” For secure with LDAPS communication, sample ldap url value is “ldaps://ad.hortonworks.COM:636” In order to configure Ranger Usersync to use STARTTLS instead of LDAPS - Add “ranger.usersync.ldap.starttls” with value “true” in custom usersync site properties in Ambari Set LdapURL property to be in the format “ldap://10.10.10.10:389” ... View more

Configuring Ranger Usersync with AD/LDAP is highly dependent on the customer environment. This requires understanding of the customer environment as well as the specific requirements for syncing users and groups. In order to get some background on User management in Hadoop, please refer to this Dataworks Summit Presentation. Corresponding slides are available at: Slides In this article we will take a common use case and see how Ranger Usersync can be configured in that scenario. Ranger Usersync supports various configuration options with AD/LDAP and details are available here Use case: Sync all the users who are members of a specific group(s) This use case can be handled in multiple ways and is also based on the AD/LDAP attributes available in the server. In case of Active Directory, users contain information of the group(s) they belong to using the “memberof” attribute. Similarly groups contain information about the user(s) that belong to them using the “member” attribute. Whereas in default installation of openLdap server doesn’t contain “memberof” attribute for the user. So the only way to retrieve user that belong to a group is by using “member” attribute of the group. Let’s take an example in Active directory where we want to sync all the users that belong to groups - “hdp_testing”, “hdp_admin”, or “dev_ops” ------------------------------------------------------------------------------------------ Sample ldapsearch command to search a particular group (hdp_testing) and determine what attributes are available for the group: ldapsearch -x -LLL -h 10.10.10.10:389 -D 'cn=administrator,CN=Users,dc=hortonworks,dc=com' -W -b 'ou=Hadoop Groups,dc=hortonworks,dc=com' 'cn=hdp_testing’ Enter LDAP Password: dn: CN=hdp_testing,ou=Hadoop Groups,dc=hortonworks,dc=com objectClass: top objectClass: group cn: hdp_testing member: CN=test11 test,ou=Hadoop Users,dc=hortonworks,dc=com member: CN=user12 user,ou=Hadoop Users,dc=hortonworks,dc=com member: CN=user8 user,ou=Hadoop Users,dc=hortonworks,dc=com member: CN=user7 user,ou=Hadoop Users,dc=hortonworks,dc=com member: CN=user6 user,ou=Hadoop Users,dc=hortonworks,dc=com member: CN=user1 user,ou=Hadoop Users,dc=hortonworks,dc=com distinguishedName: CN=hdp_testing,ou=Hadoop Groups,dc=hortonworks,dc=com instanceType: 4 uSNChanged: 1312771 name: group5 sAMAccountName: hdp_testing sAMAccountType: 222431111 Above output shows all the available attributes for hdp_testing. Note:- Highlighted are the attributes that are interested for usersync configuration. In this case hdp_testing has multiple members (highlighted “member” attribute) like member: CN=test11 test,ou=Hadoop Users,dc=hortonworks,dc=com member: CN=user12 user,ou=Hadoop Users,dc=hortonworks,dc=com member: CN=user8 user,ou=Hadoop Users,dc=hortonworks,dc=com member: CN=user7 user,ou=Hadoop Users,dc=hortonworks,dc=com etc… Sample ldapsearch command to search a particular user (user8) and determine what attributes are available for the user: ldapsearch -x -LLL -h 10.10.10.10:389 -D 'cn=administrator,CN=Users,dc=hortonworks,dc=com' -W -b 'ou=Hadoop Users,dc=hortonworks,dc=com' 'samaccountname=user8' Enter LDAP Password: dn: CN=user8 user,ou=Hadoop Users,dc=hortonworks,dc=com objectClass: top objectClass: person objectClass: organizationalPerson objectClass: user cn: user8 user sn: user givenName: user8 distinguishedName: CN=user8 user,ou=Hadoop Users,dc=hortonworks,dc=com instanceType: 4 memberOf: CN=hdp_testing,ou=Hadoop Groups,dc=hortonworks,dc=com memberOf: CN=dev_ops,ou=Hadoop Groups,dc=hortonworks,dc=com memberOf: CN=test_groups,ou=Hadoop Groups,dc=hortonworks,dc=com memberOf: CN=security_groups,ou=groups,dc=hortonworks,dc=com uSNChanged: 973651 userAccountControl: 512 badPwdCount: 0 codePage: 0 countryCode: 0 badPasswordTime: 0 lastLogoff: 0 lastLogon: 0 pwdLastSet: 0 primaryGroupID: 513 logonCount: 0 sAMAccountName: user8 Above output shows all the available attributes for a user. Note:- Highlighted are the attributes that are interested for usersync configuration. In this case user8 is a “memberof” 4 groups - hdp_testing, dev_ops, test_groups, and security_groups As you can see, users from hdp_testing, dev_ops, or hdp_admins can be sync’d to ranger by performing a user search using the “memberof” attribute in the user search filter -- User based search or a group search using the “member” attribute -- Group based search Ranger Usersync configuration contains three sets of configuration: Common Configs: Ldap Url and bind credentials User Configs: Attributes related to user like sAMAccountName, OU(s) of the users, user search filter, memberof, etc… Group Configs: Attributes related to group like sAMAccountName, OU(s) of the groups, group search filter, member, etc… Following is the screenshot of the Common Configs properties configured for the above example: Please note the following: From HDP 2.6 onwards Ranger Usersync supports “Incremental Sync” and is enabled by default. For clusters that are upgraded from older version to 2.6, “Incremental Sync” is disabled. When “Incremental sync” is enabled, “Enable Group Sync” is set to “true” by default and the properties under “Group Configs” is mandatory. Configuring Ranger Usersync for User based search: ============================================ Here Users are searched based on the attributes available from the user attribute and group information is available as part of the “memberof”. In this case two main parts that need to be configured in Ranger Usersync configuration are the properties under - “Common Configs” and “User Configs”. Optionally properties under “Group Configs” can be configured in cases where customers want to limit or filter the groups that these users belong to. As you can see in the above example, user8 belongs to 4 groups - hdp_testing, dev_ops, test_groups, and security_groups. But if we are interested only in groups hdp_testing and dev_ops, then “Group Configs” properties can be set accordingly. Following are the screenshots of User Configs and Group Configs properties configured for the above example with User based search: User Configs: ------------------------------- Group Configs: ------------------------------- Configuring Ranger Usersync for Group based search: ============================================ In this case Ranger Usersync performs group search first based on the group configuration. Here users are searched based on the attributes available from the group attributes and user information is available as part of the “member”. In this case two main parts that need to be configured in Ranger Usersync configuration are the properties under - “Common Configs” and “Group Configs”. Optionally properties under “User Configs” can be configured in cases where customers want to limit or filter the users that belong to these groups. Following are the screenshots of Group Configs, and User Configs properties configured for the above example with Group based search: Group Configs: ------------------------------- Note:- “Enable Group First Search” must be set to “true” in order to perform Group based search. User Configs: ------------------------------- Please note the following: “Enable User Search” is set to false by default. When “Enable User Search” is set to “false” none of the properties under “User Configs” will be used by Ranger Usersync Users are sync’d using group’s member attribute which in many cases contains CN (Firstname Lastname) of the user. In order to sync users using sAMAccountName, then “Enable User Search” must be set to “true” and “Username Attribute” must be set to “sAMAccountName”. Note:- With “Enable User Search” set to true, it is still required to set all other user configs appropriately like the user search base and user object class. User search filter can be set to “sAMAccountName=*” ... View more

Many deployments in hadoop use Centrify LDAP proxy to integrate Active Directory users with linux accounts. Centrify ldapproxy uses posix properties like uid, gidNumber, uidNumber, memberUid, etc... and are made available for ldap search attributes to the clients like Apache Ranger. Ranger interacts with Centrify ldapproxy for configuring hadoop authorization policies as well as for authentication to ranger UI. In order to sync users and groups from Centrify, ranger performs ldap search against Centrify using all the configured LDAP attributes. Some of the common issues noticed during this integration can be: Issue #1. Users are unable to login to Ranger UI but were successfully sync'd to ranger: For AD/LDAP authentication, Ranger uses the complete DN of the login user for performing an LDAP search. Ranger also uses group search base and group search filter for group membership for obtaining user role information. Few configuration parameters to verify on ranger are: ranger.ldap.user.dnpattern - This value should match the exact DN pattern of the login user Group Search Base - This value should be the search base of the groups which contains the login users as members Group Search Filter - This value can be a simple filter like "cn=*" Let's look at one small example: ========================================= Ldapsearch with Centrify for one of the user contains following information: dn: cn=alice bob,ou=Users,ou=us,ou=Contractor,dc=apache,dc=org description: c001038@apache.org email: abob@apache.org gecos: Alice Bob gidNumber: 422222198 homeDirectory: /home/c001038 loginShell: /bin/bash uid: c001038 uidNumber: 42222045 userPassword:: WWIEphZ2FkZWVzYW4= accountExpires: 131470704000000000 badPasswordTime: 131140254848671173 badPwdCount: 0 cn: Alice Bob codePage: 0 countryCode: 0 displayName: Alice Bob distinguishedName: CN=Alice Bob,OU=Users,OU=us,OU=C ontractor,DC=apache,DC=org givenName: Alice instanceType: 4 lastLogoff: 0 lastLogon: 131140358889710142 logonCount: 185 mail: Abob@hortonworks.com manager: CN=Joe Dave,OU=Corporate,OU=Users,OU=us,DC=apache,DC=org memberOf: CN=hdp_admin,OU=Hadoop,OU=Applications,OU=Groups,DC=apache,DC=org memberOf: cn=hdp_group1,ou=Hadoop,ou=Groups,dc=apache,dc=org name: Alice Bob objectClass: top objectClass: posixaccount objectGUID:: W6rekfmZXECSiIgFeOTnPw== objectSid:: AQUAAAAAAAUVAAAAYKiAHXLIYdQjI9RcxRQCAA== primaryGroupID: 500 pwdLastSet: 131123031899439530 sAMAccountName: c001038 sAMAccountType: 805306300 sn: Bob uSNChanged: 153583075 uSNCreated: 151863255 userAccountControl: 510 userPrincipalName: c001038@apache.org whenChanged: 20160716210250.0Z whenCreated: 20160705200144.0Z As you can observe the complete DN of the user is configured as “dn: cn=alice bob,ou=Users,ou=us,ou=Contractor,dc=apache,dc=org”. But say the user is using uid (c001038) for logging in to ranger admin and the “ranger.ldap.user.dnpattern” is configured as “uid={0},ou=Users,ou=us,ou=Contractor,dc=apache,dc=org”. In this case the ranger login fails because of the mismatch DN pattern. In order to have a successful login with uid, then Centrify config has to be changed to match the DN to be “uid=c001038,ou=Users,ou=us,ou=Contractor,dc=apache,dc=org”. Otherwise, the user can use cn (alice bob) as the login user and “ranger.ldap.user.dnpattern” can be configured as “cn={0},ou=Users,ou=us,ou=Contractor,dc=apache,dc=org” Issue #2. Groups not sync'd to Ranger: In order to provide flexibility and support different deployment scenarios, ranger provides multiple ways to configure group sync. 1. Sync groups from User’s Group name attribute (like “memberof” or “ismemberof”) - One of the use case here is to “Get all the users from the configured user search base and user search filter and get all the groups that the users belong to” In this case “Group User Map Sync” is set to “true” and “Enable Group Sync” is set to false. The only way ranger gets the group information is by using the User’s Group name attribute (which is generally “memberof” or “ismemberof” attribute of the user). Since Centrify ldapproxy uses posix attributes, most of the times “memberof” or “ismemberof” attribute is not available for the user. In this case groups will not be sync’d to Ranger. There are two ways to solve this issue - a. Without any changes to Centrify, Ranger can be configured with “Enable Group Sync” option (#2 below). b. Modify Centrify to populate “memberof” attribute for the user as in example above (memberOf:cn=hdp_admin,ou=Hadoop,ou=Applications, ou=Groups,dc=apache,dc=org memberOf: cn=hdp_group1,ou=Hadoop,ou=Groups,dc=apache,dc=org) 2. Sync groups by computing group membership of the users - One of the use case here is to “Get all the users from the configured user search base and user search filter and get only particular groups that these users belong to”. This also targets the case where LDAP server doesn’t support “memberof” attribute for the user. In this case “Group User Map Sync” should be set to “true” and “Enable Group Sync” is also should be set to true. In this case ranger first syncs all the users based on the “User Info” configuration. Then for each user ranger computes the group memberships based on the “Group Member Attribute” (like member, memberUID, etc…). Some of the common scenarios where the groups fail to sync to ranger are: a. When “Group User Map Sync” or “Enable Group Sync” not set to true b. When Group search base or group search filter are misconfigured and no matching groups are found. c. When “username” that is configured for User search is not matching the value for configured Group Member Attribute. For this let’s take an example as below: ==================================== Let’s say “User name Attribute” is configured as “uid” and “Group Member Attribute” is configured as “member” in Ranger. Following is the ldapsearch output for a group from Centrify: dn: hdp_group1,ou=Hadoop,ou=Groups,dc=apache,dc=org cn: hdp_group1 gidNumber: 478287198 memberUid: bdave memberUid: c001038 memberUid: c001140 memberUid: djoe groupType: -2122223646 member: <GUID=0b01714af9d97047aa0d55d813a20385>;<SID=0105000000000005150000006 0a8801d72c861d42323d45ce7170200>;CN=Dave Joe,OU=Users,OU=us,OU=Contractor,DC=apache,DC=org member: <GUID=3fef7ee4aa1fd84092fce8d25cfed798>;<SID=0105000000000005150000006 0a8801d72c861d42323d45ca5170200>;CN=bob dave,OU=Users,OU=us,DC=apache,DC=org member: <GUID=5baade91f9995c409288880578e4e73f>;<SID=0105000000000005150000006 0a8801d72c861d42323d45cc5140200>;CN=Alice Bob,OU=Users,OU=us,OU=C ontractor,DC=apache,DC=org objectClass: top objectClass: posixgroup sAMAccountName: hdp_grp1 uSNChanged: 152222220 Since “User name Attribute” in ranger is configured to uid, the users are sync’d to ranger using this attribute (like “c001038”). And for computing group memberships for this user, ranger uses “Group Member Attribute” which is configured as “member”. Ranger performs ldap search to get all the group configured in the group search base that contains the “member” as “uid=c001038”. But as you can see the “member” attribute returned from Centrify doesn’t contain info with uid of the user. Hence the no groups are sync’d to ranger. This issue can be fixed by configuring “Group Member Attribute” in ranger as “memberuid” so that it matches with the value of uid. 3. Sync groups from the configured search base and group search filter - One of the use case here is to “Get all the groups from the configured group search base and group search filter”. This also targets the use case to sync groups with no members. In this case“Group User Map Sync”, “Enable Group Sync” and “Enable Group Search First” should be set to true. The only reason that the groups are not sync’d to ranger because of the misconfiguration of Group search base or group search filter. Issue #3: Issues with Syncing ranger users and groups with centrify AD: Ranger retrieves only the values of the attributes configured in ranger. Hence for syncing users and/or groups ranger performs ldap search against the server (like Centrify) with all these attributes. When users and/or groups not sync’d to ranger, perform ldap search using ldapsearch utility with any particular user and/or group and with the configured attributes. Sometimes a particular attribute (like “memberof”, “member”, etc…) are not configured in Centrify and hence the user sync fails. In this case following error can be found in ranger usersync logs (/var/log/ranger/usersync/usersync.log): javax.naming.directory.NoSuchAttributeException: [LDAP: error code 16 - cdcLdapSearch :No such attribute (cdcRC=8), errSystem=Ldap, errCode=16, errString=No such attribute]; In general since Centrify uses posix attributes, “member” attribute might not be configured for a group. Instead the users are mapped to groups using “memberuid” attribute. In order to solve this issue, verify the “Group Member Attribute” is configured accordingly (memberuid in this case) in Ranger. ... View more