HBase client failing to connect to Kerberized HBase

[#1]

Here is the stacktrace:

Exception in thread "main" java.io.IOException: Login failure for hbase@EXAMPLE.COM from keytab /etc/security/keytabs/hbase.service.keytab at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1103) at org.apache.hadoop.security.UserGroupInformation$loginUserFromKeytabAndReturnUGI$0.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133) at hbase_test.run(hbase_test.groovy:23) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1215) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1024) at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:923) at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:906) at org.codehaus.groovy.runtime.InvokerHelper.runScript(InvokerHelper.java:410) at org.codehaus.groovy.runtime.InvokerHelper$runScript.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133) at hbase_test.main(hbase_test.groovy) Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856) at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719) at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687) at javax.security.auth.login.LoginContext.login(LoginContext.java:595) at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1092) ... 21 more

And this is the relevant part of the client:

Configuration conf = HBaseConfiguration.create(); conf.set("hadoop.security.authentication", "Kerberos"); UserGroupInformation.setConfiguration(conf) def userInfo = UserGroupInformation.loginUserFromKeytabAndReturnUGI("hbase@EXAMPLE.COM", args[0]);

UserGroupInformation.setLoginUser(userInfo)

[#2 · Accepted solution]

Which user did you use to run the code ?

What's the output of the following command ?

klist -kt /etc/security/keytabs/hbase.service.keytab

Normally hbase.service.keytab should be used by user 'hbase'.

Please illustrate your use case in more detail.

Please take a look at

hbase-common/src/main/java/org/apache/hadoop/hbase/AuthUtil.java

[#3]

Thanks for the klist suggestion, Ted. That and mqureshi's comment solved it for me.

[#4]

@Mike Thomsen

hbas user is usually hbase/_HOST@REALM.COM. I don't see the host part of the principal. Is this how you have setup your hbase principal?

What are the permissions on your /etc/security/keytabs/hbase.service.keytab file?

[#5]

And you were correct! It's on an AWS cluster and part of the problem was that the principle had the internal interface associated with it so even when I added what I thought was the right host, didn't work until I followed Ted's advice to use klist.

[#6]

Hi,

I am using below principals for hbase kerberos authentication:

hbase.zookeeper.quorum=localhost
hbase.zookeeper.property.clientPort=2181
hadoop.security.authentication=kerberos
hbase.security.authentication=kerberos
hbase.master.kerberos.principal=zookeeper/localhost@EXAMPLE.COM
hbase.regionserver.kerberos.principal=zookeeper/localhost@EXAMPLE.COM
hbase.kerberos.principal=zookeeper/localhost@EXAMPLE.COM
hbase.kerberos.keytab=zkpr.keytab

Now, when i run my spark job on local it is not connecting to hbase, it shows error message:

Unable to connect to zookeeper/localhost@EXAMPLE.COM to zookeeper/localhost@EXAMPLE.COM.

I have done kinit zookeeper/localhost@EXAMPLE.COM -k -t zkpr.keytab and it is running fine.

Any help will be appreciated.