Auth-to-local Rules Syntax

[#1]

Each rule in the auth-to-local rules as the following formats:

RULE:[n:string](regexp)s/pattern/replacement/
RULE:[n:string](regexp)s/pattern/replacement/g
RULE:[n:string](regexp)s/pattern/replacement//L
RULE:[n:string](regexp)s/pattern/replacement/g/L

[n:string]

Indicates a matching rule where n declares the number of expected components in the principal. Components are separated by a /, where a user account has one component (ambari-qa) and a service account has two components (nn/fqdn). The string value declares how to reformat the value to be used in the rest of the expression. The placeholders are as follows:

$0 - realm

$1 - 1st component

$2 - 2nd component

Typically we ignore the 2nd component since it is the service’s hostname and thus the format is generally set to $1@$0 (but can be any pattern) as in:

[1:$1@$0]

Matches on [email protected]

Translates to [email protected]

[2:$1@$0]

Matches on nn/[email protected]

Translates to [email protected]

(regexp)

Indicates a matching rule on the value generated by the [n:string] clause. If this regular expression (regexp) matches, then the replacement expression is invoked.

For Example:

(.*@EXAMPLE.COM)

Matches on

• [email protected]
• nn/[email protected]
• [email protected]

Does not match on

• [email protected]
• nn/[email protected]

([email protected])

Matches on

• [email protected]
• [email protected]

Does not match on

• [email protected]
• [email protected]
• [email protected]

s/pattern/replacement/

s/pattern/replacement/g

The replacement expression to use to generate a value that is to be used as the local user account. This expression is similar to (if not the same as) a sed replacement expression and is executed over the value generated by [n:string]. The pattern part of this expression is a regular expression used to find the portion of the string to replace. The replacement part of this expression is the value to use for replacing the matched section. If g is specified after the last /, the replacements will occur for every match in the value, else only the first match is processed.

For Example:

s/@.*//

Removes all characters in the source string including and after the @.

• If the source string is [email protected], the result is ambari-qa
• If the source string is [email protected], the result is any_user

s/@.*/user/

Replaces all characters in the source string including and after the @ with "user"

• If the source string is [email protected], the result is ambari-qa@user
• If the source string is [email protected], the result is any_user@user

s/abc/123/

Replaces the first substring of "abc" the source string with "123"

• If the source string is [email protected], the result is [email protected]
• If the source string is [email protected], the result is [email protected]

s/abc/123/g

Replaces all substrings of "abc" the source string with "123"

• If the source string is [email protected], the result is [email protected]
• If the source string is [email protected], the result is [email protected]

The pattern part of the expression may include capturing groups that can be reused in the replacement part of the expression. Capturing groups are declared parentheses and the data capture can be used by referencing it by number (in order of placement in the pattern). The placeholder for captured data is specified using a dollar sign and the reference number. For example $1.

s/(\d+)@.*/ID.$1/

Captures all sequences of numbers and appends "ID." to it

• If the source string is [email protected], the result is ID.1234567890

s/(\d+)([a-zA-Z]+)@.*/$2$1/

Captures all sequences of numbers and then all sequences of letters and places the letters before the numbers

• If the source string is [email protected], the result is abc123

/L

By default, translations based on rules are done maintaining the case of the input principal. For example, given the rule

RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*//

• If the source string is [email protected], the result is ambari-qa
• If the source string is [email protected], the result is AMBARI-QA
• If the source string is [email protected], the result is Ambari-QA

However this may not be desired given how different operating system handle usernames, where as some are case-sensitive and some are case-insensitive. For example, Linux is case-sensitive and Windows is case-insensitive.

To help with this issue, it is possible to force the translated result to be all lower case. This is done by adding a "/L" to the end of the rule. However, it must be noted that this does not effect how pattern matches on input and therefore that will still be case-sensitive.

RULE:[1:$1@$0](ambari-qa-.*@EXAMPLE.COM)s/.*/AMBARI-QA//L
RULE:[1:$1@$0](AMBARI-QA-.*@EXAMPLE.COM)s/.*/AMBARI-QA-UPPER//L
RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*///L

• If the source string is [email protected], the result is ambari-qa
• If the source string is [email protected], the result is ambari-qa-upper
• If the source string is joe_[email protected], the result is joe_user
• If the source string is JOE_USER@EXAMPLE.COM, the result is joe_user

Examples

RULE:[1:$1@$0](.*@HDP01.LOCAL)s/@.*//

• [email protected] → jqpublic
• [email protected] → jdoe
• nn/[email protected] → [not processed]
• dn/[email protected] → [not processed]

RULE:[1:$1@$0](.*@HDP01.LOCAL)s/.*/ambari-qa/

• [email protected] → ambari-qa
• [email protected] → ambari-qa
• nn/[email protected] → [not processed]
• dn/[email protected] → [not processed]

RULE:[2:$1@$0](.*@HDP01.LOCAL)s/@.*//

• [email protected] → [not processed]
• [email protected] → [not processed]
• nn/[email protected] → nn
• dn/[email protected] → dn

RULE:[2:$1@$0](.*@HDP01.LOCAL)s/.*/hdfs/

• [email protected] → [not processed]
• [email protected] → [not processed]
• nn/[email protected] → hdfs
• dn/[email protected] → hdfs

Rule Processing

When processing auth-to-local rules, each rule in the ruleset is processed in order. When a match is made, the processing routine effectively exits and returns the translation that was generated.

For example, if the rule set was:

RULE:[1:$1@$0]([email protected])s/.*/hdfs/
RULE:[1:$1@$0](.*@EXAMPLE.COM)s/.*/not_hdfs/

• [email protected] would yield "hdfs"
• [email protected] would yield "not_hdfs"

However, if the ruleset was:

RULE:[1:$1@$0](.*@EXAMPLE.COM)s/.*/not_hdfs/
RULE:[1:$1@$0]([email protected])s/.*/hdfs

• [email protected] would yield "not_hdfs"
• [email protected] would yield "not_hdfs"

Testing Rulesets

Since auth-to-local rulesets can be rather difficult to read and determine correctness, a handy tool can be used to test it out. However this tool reads the ruleset from the hadoop.security.auth_to_local property in the core-site.xml file (typically found at /etc/hadoop/conf/core-site.xml) and may not be able to import rules from a different source.

To use the tool, one of two commands can be executed on the command line:

Newer versions of hadoop should use:

hadoop kerbname

Older versions of hadoop should use:

hadoop org.apache.hadoop.security.HadoopKerberosName

For example:

hadoop org.apache.hadoop.security.HadoopKerberosName [email protected]
Name: [email protected] to joe_user

hadoop org.apache.hadoop.security.HadoopKerberosName [email protected]
Name: [email protected] to ambari-qa

[#2]

Saved that as PDF. I always wanted to look up how they work but followed through. Thanks a lot.

[#3]

If any rule matches, does it stop processing other rules?

Stop or not wouldn't be issue in most of the case but in case someone put completely wrong rules...

[#4]

@Hajime, that would be worth adding to this document. Thanks for asking.

The rules are processed in order from top to bottom. When a match is found, processing stops.

Therefore with

RULE:[1:$1@$0]([email protected])s/.*/hdfs/
RULE:[1:$1@$0](.*@EXAMPLE.COM)s/.*/not_hdfs/

• [email protected] would yield "hdfs"
• [email protected] would yield "not_hdfs"

And with

RULE:[1:$1@$0](.*@EXAMPLE.COM)s/.*/not_hdfs/
RULE:[1:$1@$0]([email protected])s/.*/hdfs

• [email protected] would yield "not_hdfs"
• [email protected] would yield "not_hdfs"

[#5]

Great article @Robert Levas !

For the completeness purpose, I'd like to see this information added to the above article:

While mapping the Kerberos principals, if the Kerberos principal names are in the UPPER case or CaMeL case, they won't be recognized on the Linux machine (as Linux users are always in lower case). So you need to add the extra switch "/L" in the RULE definition to force the conversion to lower case.

For example, here are some auth_to_local rule examples with lower case switch added:

"RULE:[1:$1]/L"
"RULE:[2:$1]/L"
"RULE:[2:$1;$2](^.*;admin$)s/;admin$///L"
"RULE:[2:$1;$2](^.*;guest$)s/;guest$//g/L"

And based on these rules, here are the expected output for the following inputs:

"[email protected]" to "joe"
"Joe/[email protected]" to "joe"
"Joe/[email protected]" to "joe"
"Joe/[email protected]" to "joe"

Hope this helps.

[#6]

@Robert Levas, thanks for the great article! May I also suggest adding information about the "hadoop kerbname" or "hadoop org.apache.hadoop.security.HadoopKerberosName" shell command? This is a helpful debugging tool that prints the current prinicipal's short name after Hadoop applies the currently configured auth_to_local rules. If you'd like, feel free to copy-paste my text from this answer: https://community.hortonworks.com/questions/38573/pig-view-hdfs-test-failing-service-hdfs-check-fail... .

[#7]

@Chris Nauroth

The hadoop kerbname tool is awesome! I just had a opportunity to use it to test out a complex rule, but I needed to use the _old_ syntax to get it to work (thanks for referencing your comment that explains the usage):

hadoop org.apache.hadoop.security.HadoopKerberosName [email protected]

I meant to add this to the doc, I just haven't had the time. I will get around to it though. But thanks for proving the tool.

[#8]

Very helpful, thanks!

[#9]

I have a use case where given a rule like:

RULE:[1:$1@$0](.*@FOO.COM)s/@.*// 

want *all* of the below username's to match

[email protected] 
[email protected] 
[email protected] 

I tried modifying the above rule to use the RegEx modifier that makes the (regexp) case insensitive:

RULE:[1:$1@$0]((?i)(.*@FOO.COM))s/@.*// 

However on testing with this rule I get an error :

$ hadoop org.apache.hadoop.security.HadoopKerberosName  [email protected]
Exception in thread "main" java.util.regex.PatternSyntaxException: Unknown inline modifier near index 3
(?i
   ^
        at java.util.regex.Pattern.error(Pattern.java:1957)
        at java.util.regex.Pattern.group0(Pattern.java:2896)
        at java.util.regex.Pattern.sequence(Pattern.java:2053)
        at java.util.regex.Pattern.expr(Pattern.java:1998)
        at java.util.regex.Pattern.compile(Pattern.java:1698)
        at java.util.regex.Pattern.<init>(Pattern.java:1351)
        at java.util.regex.Pattern.compile(Pattern.java:1028)
        at org.apache.hadoop.security.authentication.util.KerberosName$Rule.<init>(KerberosName.java:193)
        at org.apache.hadoop.security.authentication.util.KerberosName.parseRules(KerberosName.java:342)
        at org.apache.hadoop.security.authentication.util.KerberosName.setRules(KerberosName.java:398)
        at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:75)
        at org.apache.hadoop.security.HadoopKerberosName.main(HadoopKerberosName.java:79)

Any idea what I'm doing wrong ?

[#10]

@Anant Aneja

You probably should have posed this as a question in the form, rather than a comment to this article. It may have gotten answered quicker.

The rule you are using will not perform the translation you want. The regular expression syntax to match using case-insensitivity is not supported as you have specified it and the translation will not generate local names with all lower-case characters. The rule you want is more like

RULE:[1:$1@$0](.*@FOO.COM)s////L

With this rule, the Hadoop UGI class will translate [email protected] to [email protected]

[root@c7401 ~]# hadoop org.apache.hadoop.security.HadoopKerberosName [email protected]
18/08/27 15:57:07 INFO util.KerberosName: Non-simple name [email protected] after auth_to_local rule RULE:[1:$1@$0](.*@FOO.COM)s////L
Name: [email protected] to [email protected]

As for the other principal names, they will technically be invalid since the realm name needs to always be in all upper-case characters.

• [email protected] - legal
• [email protected] - illegal
• [email protected] - illegal

[#11]

Hi team,
I want to configuration

"yarn-user/[email protected]" to "yarn-user"
"yarn-user/[email protected]" to "yarn-user"
"yarn-user/[email protected]" to "yarn-user"

Please given a rule advidor.
Thanks