Yarn MR overloads Active Directory domain controller CPU

[#1]

Hi,

I have setup our Clouder cluster with Kerbers + AD.

A user authenticate through MIT Kerberos to AD, and user/group info is read through ShellBasedUnixGroupsMapping to AD.

If we run MR job with 300 simultaneous mappers (total 7000 mappers), our domain controller in AD hits CPU load of 100% throughout the job execution.

I executed MR job using Hive after appropritately authenticating with Kerberos server (kinit).

Why would MR job cause such CPU load?

My initial thought was MR authenticate and gets user/group info once when job starts or hive-shell stats. Apparently my job is constantly overloading AD resources.

Does the job casue CPU spike each time mapper gets created? or is it each time mapper access HDFS?

And what would be the best way to resolve this issue?

Best regards,

Ben

[#2 · Accepted solution]

Have you or your AD admins also attempted to profile what specific AD operation(s) are pouring in? Are they group lookups? Or are they actual authentication requests? The latter would normally be unexpected, given use of tokens won't require re-auth.

Group lookups are indeed done for every HDFS operation when permissions are in use. However, the groups are also cached internally by HDFS for 5m by default (configurable), and often also by a NameNode-local NSCD/equivalent service. These things help reduce the backend load, but the need is certainly present and the cache timeouts are finite, so it wouldn't be too odd to see a lot of group related requests get fired to whatever user directory backend is in use.

Are you already using NSCD? Perhaps that may help you if you aren't, or you can consider raising HDFS's cache timeout: http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-common/core-default.xml#had...

[#3]

Hi Harsh

I was thinking of configuring separate NIS server for user/group mapping, however you came in at right time to save me 😄

NSCD in fact resolved the issue. NSCD was turned off on chkconfig by default, we switched it on, now no more heavy CPU load on dc servers.

Now I wonder why hadoop.security.groups.cache.secs didn't have effect from the beginning?

We haven't specifically configured it on our system, but isn't it set to 300sec by default?

Harsh, i know you helped me couple times in the past, thank you very much for your support!

Ben

[#4]

The security group cache will only work within a JVM. If you have lots of JVM's or short lived JVM's for your jobs then caching inside the JVM will only give a limited relief.

NSCD will prevent the OS going out for every call that is made and works over different JVM's. So instead of having one call per JVM you will now have one call for a lot of JVM's

Wilfred

[#5]

To add onto Wilfred's response, what is your CDH version?

HDFS does cache all positive entries for 5 minutes, but negative caching wasn't supported until CDH 5.2.0 onwards (via HADOOP-10755). See also http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-common/core-default.xml#had... (which lists negative caching's TTL default being 30s, vs. positive's 300s).

NSCD does also do negative caching by default, which could explain why the problem is gone, depending on how many negative, WARN group-lookup failure entries you observe in the log.