After enabling kerberos, using hbase through java (using ycsb here for bencharking), unable to login to hbase and getting response as "Illegal principal name" and "No rules applied to"

[#1]

I have a kerberized cluster where in local realm trusts AD realm with MIT KDC setup.

AD Realm : EXAMPLE.COM

Local Realm: LOCALREALM.EXAMPLE.COM

Post doing kinit as user@EXAMPLE.COM , I'm able to perform all the regular tasks through command line like creating hbase tables, running mapreduce job etc.

But, when i'm trying to connect to hbase to perform a benchmarking through ycsb tool, it throws an exception as unable to login.

If i authenticate using the local realm such as user@LOCALREALM.EXAMPLE.COM, it works like a charm.

I have the rules added in auth to local to trust AD realm too : RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*//

Do not understand if i'm missing anything else. can someone please help ?

Below is a part of the stack trace:

Caused by: java.io.IOException: failure to login

at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:782)

at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:734)

at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:607)

at org.apache.hadoop.hbase.security.User$SecureHadoopUser.<init>(User.java:285)

at org.apache.hadoop.hbase.security.User$SecureHadoopUser.<init>(User.java:281)

at org.apache.hadoop.hbase.security.User.getCurrent(User.java:185)

at org.apache.hadoop.hbase.security.UserProvider.getCurrent(UserProvider.java:88)

at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:215)

at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)

at com.yahoo.ycsb.db.HBaseClient10.init(HBaseClient10.java:149)

... 3 more

Caused by: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name user@EXAMPLE.COM

at org.apache.hadoop.security.User.<init>(User.java:50)

at org.apache.hadoop.security.User.<init>(User.java:43)

at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:179)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)

at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)

at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)

at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)

at javax.security.auth.login.LoginContext.login(LoginContext.java:588)

at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:757)

at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:734)

at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:607)

at org.apache.hadoop.hbase.security.User$SecureHadoopUser.<init>(User.java:285)

at org.apache.hadoop.hbase.security.User$SecureHadoopUser.<init>(User.java:281)

at org.apache.hadoop.hbase.security.User.getCurrent(User.java:185)

at org.apache.hadoop.hbase.security.UserProvider.getCurrent(UserProvider.java:88)

at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:215)

at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)

at com.yahoo.ycsb.db.HBaseClient10.init(HBaseClient10.java:149)

at com.yahoo.ycsb.DBWrapper.init(DBWrapper.java:86)

at com.yahoo.ycsb.ClientThread.run(Client.java:424)

at java.lang.Thread.run(Thread.java:748)

Caused by: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to user@EXAMPLE.COM

at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389)

at org.apache.hadoop.security.User.<init>(User.java:48)

... 26 more

[#2 · Accepted solution]

@Narendra Neerukonda

I am not familiar with the ycsb tool. This error coming from that tool. Does the ycsb tool have the relevant auth-to-local rules configured?

[#3]

ycsb is a standalone tool. We need to pass the hbase-site.xml and we can run the tests (benchmarking tool for databases)

I'm trying to figure out if any specific auth-to-local rules are required to be configured in ambari. Since i'm triggering it with my user id after authenticating with AD realm (AD realm added to auth-to-local rules), not able to understand why i still have the error.

As far as i'm understanding, the error is not originating from the tool as i'm able to use/run ycsb benchmarking if i authenticate using the local realm (i added my user principal to the local MIT kdc and authenticated using that----getting a ticket as user@LOCALREALM.EXAMPLE.COM instead of user@EXAMPLE.COM).

when using kinit as user@EXAMPLE.COM and running, i'm getting below responses as in the above stack trace:

Caused by: java.io.IOException: failure to login

Caused by: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name user@EXAMPLE.COM

Caused by: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to user@EXAMPLE.COM

[#4]

It seems like hbase-site.xml does not contain auth-to-local rules and that Hbase may take those rules from the core-site.xml file. That said, Ambari will add the needed rules to the core-site.xml file - hadoop.security.auth_to_local - if it known about the additional realm(s). This is done by added EXAMPLE.COM to the Additional Realms field in the Kerberos administration view - as discussed in https://community.hortonworks.com/questions/227267/unable-to-authenticate-as-username-to-cluster-aft....

Playing with this more, I am able to generate the error you are getting if the auth-to-local rules are not set up properly in core-site.xml. You can test your's my running the following command (not via ycsb):

hadoop kerbname <principal name> 

Or by running

hadoop org.apache.hadoop.security.HadoopKerberosName <principal name> 	

For example:

[root@c7401 ~]# hadoop org.apache.hadoop.security.HadoopKerberosName user@UNKNOWN.DOM
18/11/11 14:36:19 INFO util.KerberosName: No auth_to_local rules applied to user@UNKNOWN.DOM
Name: user@UNKNOWN.DOM to user@UNKNOWN.DOM

Since I do not have the full stack track or all of the information, I cannot comment on whether the ycsb tool or Hbase is generating that error. If it is Hbase, itself, then the hadoop kerbname command (on the relevant host) should show the same error when passing "user@EXAMPLE.COM" to it - assuming Hbase really does use core-site.xml to load the auth-to-local rules. However, if that command does not show the "no auth_to_local_ rules" message, then I would have to assume the error is coming from the ycsb tool and the appropriate core-site.xml file is needed.

[#5]

Got it.

It started working fine once i passed the core-site.xml properly to the tool. Seems it wasn't able to pick up the rules as it didn't read the core-site.xml file or something.

Thank you @Robert Levas for helping out.

[#6]

Awesome! I am glad that I could help out.

[#7]

Because of below parameter in core-site.xml which works for you

hadoop.security.auth_to_local

The mapping from Kerberos principal names to local OS user names. See Creating Mappings Between Principals and UNIX Usernames for more information.

[#8]

Because of this below property in core-site.xml, it works for you

hadoop.security.auth_to_local

The mapping rules. For example: RULE:[2:$1@$0]([jt]t@.*EXAMPLE.COM)s/.*/mapred/ RULE:[2:$1@$0]([nd]n@.*EXAMPLE.COM)s/.*/hdfs/ RULE:[2:$1@$0](hm@.*EXAMPLE.COM)s/.*/hbase/ RULE:[2:$1@$0](rs@.*EXAMPLE.COM)s/.*/hbase/ DEFAULT

The mapping from Kerberos principal names to local OS user names. See Creating Mappings Between Principals and UNIX Usernames for more information.