Created on 06-08-2016 06:51 PM
Hadoop may be configured to use LDAP as the source for resolving an authenticated user's list of group memberships. A common example where Hadoop needs to resolve group memberships is the permission checks performed by HDFS at the NameNode. The Apache documentation's HDFS Permissions Guide contains further discussion of how the group mapping works: the NameNode calls a configurable plugin to get the user's group memberships before checking permissions.
Despite that document's focus on group resolution at the NameNode, many other Hadoop processes also call the group mapping. The information in this document applies to the entire ecosystem of Hadoop-related components.
As described in that document, the exact implementation of the group mapping is configurable. Here is the documentation of the configuration property from core-default.xml and its default value.
<property> <name>hadoop.security.group.mapping</name> <value>org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback</value> <description> Class for user to group mapping (get groups for a given user) for ACL. The default implementation, org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, will determine if the Java Native Interface (JNI) is available. If JNI is available the implementation will use the API within hadoop to resolve a list of groups for a user. If JNI is not available then the shell implementation, ShellBasedUnixGroupsMapping, is used. This implementation shells out to the Linux/Unix environment with the <code>bash -c groups</code> command to resolve a list of groups for a user. </description> </property>
LDAP integration arises from several possible configuration scenarios:
If there is any doubt about how Hadoop is resolving a user's group memberships, then a helpful troubleshooting step is to run the following command while logged in as the user. This will print authentication information for the current user, including group memberships, as they are really seen by the Hadoop code.
> hadoop org.apache.hadoop.security.UserGroupInformation Getting UGI for current user User: chris Group Ids: Groups: staff everyone localaccounts _appserverusr admin _appserveradm _lpadmin _appstore _lpoperator _developer com.apple.access_screensharing com.apple.access_ssh UGI: chris (auth:SIMPLE) Auth method SIMPLE Keytab false ============================================================
However, in the case of HDFS file permissions, recall that the group resolution really occurs at the NameNode before it checks authorization for the user. If configuration is different at the NameNode compared to the client host, then it's possible that the NameNode will see different results for the group memberships. To see the NameNode's opinion of the user's group memberships, run the following command.
> hdfs groups chris : staff everyone localaccounts _appserverusr admin _appserveradm _lpadmin _appstore _lpoperator _developer com.apple.access_screensharing com.apple.access_ssh
As a distributed system running across hundreds or thousands of nodes, all independently resolving users' group memberships, this usage pattern may generate unexpectedly high call volume to the LDAP infrastructure. Typical symptoms are slow responses from the LDAP server, perhaps resulting in timeouts. If group resolution takes too long, then the Hadoop process may log a message like this:
2016-06-07 13:07:00,831 WARN security.Groups (Groups.java:getGroups(181)) - Potential performance problem: getGroups(user=chris) took 13018 milliseconds.
The exact timeout threshold for this warning is configurable, with a default value of 5 seconds.
<property> <name>hadoop.security.groups.cache.warn.after.ms</name> <value>5000</value> <description> If looking up a single user to group takes longer than this amount of milliseconds, we will log a warning message. </description> </property>
The exact impact to the Hadoop process varies. In many cases, such as execution of a YARN container running a map task, the delay simply increases total latency of execution for that container.
A more harmful case is slow lookup at the HDFS JournalNode. If multiple JournalNodes simultaneously experience a long delay in group resolution, then it's possible to exceed the NameNode's timeout for JournalNode calls. The NameNode must be able to log edits to a quorum of JournalNodes (i.e. 2 out of 3 JournalNodes). If the calls time out to 2 or more JournalNodes, then it's a fatal condition. The NameNode must be able to log transactions successfully, and if it fails, then it aborts intentionally. This condition would trigger an unwanted HA failover. The problem might reoccur after failover, resulting in flapping. If this happens, then the JournalNode logs will show the "performance problem" mentioned above, and the NameNode logs will show a message about "Timed out waiting for a quorum of nodes to respond" before a FATAL shutdown error.
If your cluster is encountering problems due to high load on LDAP infrastructure, then there are several possible ways to mitigate this by tuning the Hadoop deployment.
Hadoop supports in-process caching of group membership resolution data. There are several configuration properties that control the behavior of the cache. Tuning these properties may help mitigate LDAP load issues.
<property> <name>hadoop.security.groups.cache.secs</name> <value>300</value> <description> This is the config controlling the validity of the entries in the cache containing the user->group mapping. When this duration has expired, then the implementation of the group mapping provider is invoked to get the groups of the user and then cached back. </description> </property>
<property> <name>hadoop.security.groups.negative-cache.secs</name> <value>30</value> <description> Expiration time for entries in the the negative user-to-group mapping caching, in seconds. This is useful when invalid users are retrying frequently. It is suggested to set a small value for this expiration, since a transient error in group lookup could temporarily lock out a legitimate user. Set this to zero or negative value to disable negative user-to-group caching. </description> </property>
The NameNode and ResourceManager provide administrative commands for forcing invalidation of the in-process group cache. This can be useful for propagating group membership changes without requiring a restart of the NameNode or ResourceManager process.
> hdfs dfsadmin -refreshUserToGroupsMappings Refresh user to groups mapping successful
> yarn rmadmin -refreshUserToGroupsMappings 16/06/08 11:38:20 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8033
If the host OS integrates with LDAP (e.g. hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback and the host OS uses pam_ldap), then the Name Service Cache Daemon is an effective approach for caching group memberships at the OS layer. Note that this approach is superior to Hadoop's in-process caching, because nscd would allow multiple Hadoop processes running on the same host to share a common cache and avoid repeated lookups across different processes. However, nscd is unlikely to be beneficial if hadoop.security.group.mapping=org.apache.hadoop.security.LdapGroupsMapping, because Hadoop processes will issue their own LDAP calls directly instead of delegating to the host OS.
Hadoop also supports specifying a static mapping of users to their group memberships in configuration in core-site.xml.
<property> <name>hadoop.user.group.static.mapping.overrides</name> <value>dr.who=;</value> <description> Static mapping of user to groups. This will override the groups if available in the system for the specified user. In otherwords, groups look-up will not happen for these users, instead groups mapped in this configuration will be used. Mapping should be in this format. user1=group1,group2;user2=;user3=group2; Default, "dr.who=;" will consider "dr.who" as user without groups. </description> </property>
This approach completely bypasses LDAP (or any other group lookup mechanism) for the specified users. A drawback of this approach is that administrators lose centralized management of group memberships through LDAP for the specified users. In practice, this is not a significant drawback for the HDP service principals, which generally don't change their group memberships. For example:
<property> <name>hadoop.user.group.static.mapping.overrides</name> <value>hive=hadoop,hive;hdfs=hadoop,hdfs;oozie=users,hadoop,oozie;knox=hadoop;mapred=hadoop,mapred;zookeeper=hadoop;falcon=hadoop;sqoop=hadoop;yarn=hadoop;hcat=hadoop;ams=hadoop;root=hadoop;ranger=hadoop;rangerlogger=hadoop;rangeradmin=hadoop;ambari-qa=hadoop,users;</value> </property>
Static mapping is particularly effective at mitigating the problem of slow group lookups at the JournalNode discussed earlier. JournalNode calls are almost exclusively performed by the hdfs service principal, so specifying it in static mapping prevents the need for the JournalNode to call LDAP.
Any configuration tuning would require a restart of the relevant Hadoop process (such as NameNode or JournalNode) for the change to take effect.