Hadoop and LDAP: Usage, Load Patterns and Tuning

[#1]

LDAP Usage

Hadoop may be configured to use LDAP as the source for resolving an authenticated user's list of group memberships. A common example where Hadoop needs to resolve group memberships is the permission checks performed by HDFS at the NameNode. The Apache documentation's HDFS Permissions Guide contains further discussion of how the group mapping works: the NameNode calls a configurable plugin to get the user's group memberships before checking permissions.

Despite that document's focus on group resolution at the NameNode, many other Hadoop processes also call the group mapping. The information in this document applies to the entire ecosystem of Hadoop-related components.

As described in that document, the exact implementation of the group mapping is configurable. Here is the documentation of the configuration property from core-default.xml and its default value.

 <property>
  <name>hadoop.security.group.mapping</name>
  <value>org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback</value>
  <description>
    Class for user to group mapping (get groups for a given user) for ACL. 
    The default implementation,
    org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, 
    will determine if the Java Native Interface (JNI) is available. If JNI is 
    available the implementation will use the API within hadoop to resolve a 
    list of groups for a user. If JNI is not available then the shell 
    implementation, ShellBasedUnixGroupsMapping, is used.  This implementation 
    shells out to the Linux/Unix environment with the 
    <code>bash -c groups</code> command to resolve a list of groups for a user.
  </description>
</property>

LDAP integration arises from several possible configuration scenarios:

1. hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, and the host OS integrates directly with LDAP, such as via pam_ldap. A Hadoop process will look up group memberships via standard syscalls, and those syscalls will delegate to pam_ldap.
2. hadoop.security.group.mapping=org.apache.hadoop.security.LdapGroupsMapping. A Hadoop process will call the LDAP server directly. This can be useful if the host OS cannot integrate with LDAP for some reason. As a side effect, it is possible that Hadoop will see a different list of group memberships for a user compared to what the host OS reports, such as by running the "groups" command at the shell.
3. Since group mapping is pluggable, it is possible (though rare) that a deployment has configured hadoop.security.group.mapping as a custom implementation of the org.apache.hadoop.security.GroupMappingServiceProvider interface. In that case, the integration pattern will vary depending on the implementation details.

Troubleshooting Group Membership

If there is any doubt about how Hadoop is resolving a user's group memberships, then a helpful troubleshooting step is to run the following command while logged in as the user. This will print authentication information for the current user, including group memberships, as they are really seen by the Hadoop code.

> hadoop org.apache.hadoop.security.UserGroupInformation
Getting UGI for current user
User: chris
Group Ids: 
Groups: staff everyone localaccounts _appserverusr admin _appserveradm _lpadmin _appstore _lpoperator _developer com.apple.access_screensharing com.apple.access_ssh 
UGI: chris (auth:SIMPLE)
Auth method SIMPLE
Keytab false
============================================================

However, in the case of HDFS file permissions, recall that the group resolution really occurs at the NameNode before it checks authorization for the user. If configuration is different at the NameNode compared to the client host, then it's possible that the NameNode will see different results for the group memberships. To see the NameNode's opinion of the user's group memberships, run the following command.

> hdfs groups
chris : staff everyone localaccounts _appserverusr admin _appserveradm _lpadmin _appstore _lpoperator _developer com.apple.access_screensharing com.apple.access_ssh

Load Patterns

As a distributed system running across hundreds or thousands of nodes, all independently resolving users' group memberships, this usage pattern may generate unexpectedly high call volume to the LDAP infrastructure. Typical symptoms are slow responses from the LDAP server, perhaps resulting in timeouts. If group resolution takes too long, then the Hadoop process may log a message like this:

2016-06-07 13:07:00,831 WARN  security.Groups (Groups.java:getGroups(181)) - Potential performance problem: getGroups(user=chris) took 13018 milliseconds.

The exact timeout threshold for this warning is configurable, with a default value of 5 seconds.

<property>
  <name>hadoop.security.groups.cache.warn.after.ms</name>
  <value>5000</value>
  <description>
    If looking up a single user to group takes longer than this amount of
    milliseconds, we will log a warning message.
  </description>
</property>

Impacts

The exact impact to the Hadoop process varies. In many cases, such as execution of a YARN container running a map task, the delay simply increases total latency of execution for that container.

A more harmful case is slow lookup at the HDFS JournalNode. If multiple JournalNodes simultaneously experience a long delay in group resolution, then it's possible to exceed the NameNode's timeout for JournalNode calls. The NameNode must be able to log edits to a quorum of JournalNodes (i.e. 2 out of 3 JournalNodes). If the calls time out to 2 or more JournalNodes, then it's a fatal condition. The NameNode must be able to log transactions successfully, and if it fails, then it aborts intentionally. This condition would trigger an unwanted HA failover. The problem might reoccur after failover, resulting in flapping. If this happens, then the JournalNode logs will show the "performance problem" mentioned above, and the NameNode logs will show a message about "Timed out waiting for a quorum of nodes to respond" before a FATAL shutdown error.

Tuning

If your cluster is encountering problems due to high load on LDAP infrastructure, then there are several possible ways to mitigate this by tuning the Hadoop deployment.

In-Process Caching

Hadoop supports in-process caching of group membership resolution data. There are several configuration properties that control the behavior of the cache. Tuning these properties may help mitigate LDAP load issues.

<property>
  <name>hadoop.security.groups.cache.secs</name>
  <value>300</value>
  <description>
    This is the config controlling the validity of the entries in the cache
    containing the user->group mapping. When this duration has expired,
    then the implementation of the group mapping provider is invoked to get
    the groups of the user and then cached back.
  </description>
</property>

<property>
  <name>hadoop.security.groups.negative-cache.secs</name>
  <value>30</value>
  <description>
    Expiration time for entries in the the negative user-to-group mapping
    caching, in seconds. This is useful when invalid users are retrying
    frequently. It is suggested to set a small value for this expiration, since
    a transient error in group lookup could temporarily lock out a legitimate
    user.
    Set this to zero or negative value to disable negative user-to-group caching.
  </description>
</property>

The NameNode and ResourceManager provide administrative commands for forcing invalidation of the in-process group cache. This can be useful for propagating group membership changes without requiring a restart of the NameNode or ResourceManager process.

> hdfs dfsadmin -refreshUserToGroupsMappings
Refresh user to groups mapping successful

> yarn rmadmin -refreshUserToGroupsMappings
16/06/08 11:38:20 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8033

External Caching with Name Service Cache Daemon

If the host OS integrates with LDAP (e.g. hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback and the host OS uses pam_ldap), then the Name Service Cache Daemon is an effective approach for caching group memberships at the OS layer. Note that this approach is superior to Hadoop's in-process caching, because nscd would allow multiple Hadoop processes running on the same host to share a common cache and avoid repeated lookups across different processes. However, nscd is unlikely to be beneficial if hadoop.security.group.mapping=org.apache.hadoop.security.LdapGroupsMapping, because Hadoop processes will issue their own LDAP calls directly instead of delegating to the host OS.

Static Group Mapping

Hadoop also supports specifying a static mapping of users to their group memberships in configuration in core-site.xml.

<property>
  <name>hadoop.user.group.static.mapping.overrides</name>
  <value>dr.who=;</value>
  <description>
    Static mapping of user to groups. This will override the groups if
    available in the system for the specified user. In otherwords, groups
    look-up will not happen for these users, instead groups mapped in this
    configuration will be used.
    Mapping should be in this format.
    user1=group1,group2;user2=;user3=group2;
    Default, "dr.who=;" will consider "dr.who" as user without groups.
  </description>
</property>

This approach completely bypasses LDAP (or any other group lookup mechanism) for the specified users. A drawback of this approach is that administrators lose centralized management of group memberships through LDAP for the specified users. In practice, this is not a significant drawback for the HDP service principals, which generally don't change their group memberships. For example:

<property>
  <name>hadoop.user.group.static.mapping.overrides</name>
  <value>hive=hadoop,hive;hdfs=hadoop,hdfs;oozie=users,hadoop,oozie;knox=hadoop;mapred=hadoop,mapred;zookeeper=hadoop;falcon=hadoop;sqoop=hadoop;yarn=hadoop;hcat=hadoop;ams=hadoop;root=hadoop;ranger=hadoop;rangerlogger=hadoop;rangeradmin=hadoop;ambari-qa=hadoop,users;</value>
</property>

Static mapping is particularly effective at mitigating the problem of slow group lookups at the JournalNode discussed earlier. JournalNode calls are almost exclusively performed by the hdfs service principal, so specifying it in static mapping prevents the need for the JournalNode to call LDAP.

Any configuration tuning would require a restart of the relevant Hadoop process (such as NameNode or JournalNode) for the change to take effect.

[#2]

And learned some new things as well. Never knew that Hadoop can go directly to LDAP as well. Also static mapping is interesting.