question Re: HBase client failing to connect to Kerberized HBase in Support Questions

HBase client failing to connect to Kerberized HBase

MikeThomsen — Sat, 03 Sep 2016 08:12:53 GMT

Here is the stacktrace:

Exception in thread "main" java.io.IOException: Login failure for hbase@EXAMPLE.COM from keytab /etc/security/keytabs/hbase.service.keytab at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1103) at org.apache.hadoop.security.UserGroupInformation$loginUserFromKeytabAndReturnUGI$0.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133) at hbase_test.run(hbase_test.groovy:23) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1215) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1024) at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:923) at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:906) at org.codehaus.groovy.runtime.InvokerHelper.runScript(InvokerHelper.java:410) at org.codehaus.groovy.runtime.InvokerHelper$runScript.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133) at hbase_test.main(hbase_test.groovy) Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856) at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719) at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687) at javax.security.auth.login.LoginContext.login(LoginContext.java:595) at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1092) ... 21 more

And this is the relevant part of the client:

Configuration conf = HBaseConfiguration.create(); conf.set("hadoop.security.authentication", "Kerberos"); UserGroupInformation.setConfiguration(conf) def userInfo = UserGroupInformation.loginUserFromKeytabAndReturnUGI("hbase@EXAMPLE.COM", args[0]);

UserGroupInformation.setLoginUser(userInfo)

Re: HBase client failing to connect to Kerberized HBase

tyu — Sat, 03 Sep 2016 08:57:45 GMT

Which user did you use to run the code ?

What's the output of the following command ?

klist -kt /etc/security/keytabs/hbase.service.keytab

Normally hbase.service.keytab should be used by user 'hbase'.

Please illustrate your use case in more detail.

Please take a look at

hbase-common/src/main/java/org/apache/hadoop/hbase/AuthUtil.java

Re: HBase client failing to connect to Kerberized HBase

mqureshi — Sat, 03 Sep 2016 09:29:38 GMT

@Mike Thomsen

hbas user is usually hbase/_HOST@REALM.COM. I don't see the host part of the principal. Is this how you have setup your hbase principal?

What are the permissions on your /etc/security/keytabs/hbase.service.keytab file?

Re: HBase client failing to connect to Kerberized HBase

MikeThomsen — Sat, 03 Sep 2016 21:48:11 GMT

Thanks for the klist suggestion, Ted. That and mqureshi's comment solved it for me.

Re: HBase client failing to connect to Kerberized HBase

MikeThomsen — Sat, 03 Sep 2016 21:49:04 GMT

And you were correct! It's on an AWS cluster and part of the problem was that the principle had the internal interface associated with it so even when I added what I thought was the right host, didn't work until I followed Ted's advice to use klist.

Re: HBase client failing to connect to Kerberized HBase

p_hasija91 — Thu, 25 May 2017 13:09:19 GMT

Hi,

I am using below principals for hbase kerberos authentication:

hbase.zookeeper.quorum=localhost
hbase.zookeeper.property.clientPort=2181
hadoop.security.authentication=kerberos
hbase.security.authentication=kerberos
hbase.master.kerberos.principal=zookeeper/localhost@EXAMPLE.COM
hbase.regionserver.kerberos.principal=zookeeper/localhost@EXAMPLE.COM
hbase.kerberos.principal=zookeeper/localhost@EXAMPLE.COM
hbase.kerberos.keytab=zkpr.keytab

Now, when i run my spark job on local it is not connecting to hbase, it shows error message:

Unable to connect to zookeeper/localhost@EXAMPLE.COM to zookeeper/localhost@EXAMPLE.COM.

I have done kinit zookeeper/localhost@EXAMPLE.COM -k -t zkpr.keytab and it is running fine.

Any help will be appreciated.