Support Questions
Find answers, ask questions, and share your expertise

Testing kerberized HBase cluster from a development machine

Explorer

We have a cluster deployed on AWS and our development workstations are located on a corporate network. Does anyone have a recommendation on how to set up a HBase client to work in this environment? We have a client that works when run on the cluster, but running it from one of our machines results in this stacktrace:

Exception in thread "main" java.io.IOException: Login failure for USER/HOST@EXAMPLE.COM from keytab \temp\USER.keytab
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1103)
	at org.apache.hadoop.security.UserGroupInformation$loginUserFromKeytabAndReturnUGI$0.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
	at hbase_test.run(hbase_test.groovy:23)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1215)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1024)
	at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:923)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:906)
	at org.codehaus.groovy.runtime.InvokerHelper.runScript(InvokerHelper.java:410)
	at org.codehaus.groovy.runtime.InvokerHelper$runScript.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
	at hbase_test.main(hbase_test.groovy)
Caused by: javax.security.auth.login.LoginException: HOST
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Unknown Source)
	at com.sun.security.auth.module.Krb5LoginModule.login(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at javax.security.auth.login.LoginContext.invoke(Unknown Source)
	at javax.security.auth.login.LoginContext.access$000(Unknown Source)
	at javax.security.auth.login.LoginContext$4.run(Unknown Source)
	at javax.security.auth.login.LoginContext$4.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.login.LoginContext.invokePriv(Unknown Source)
	at javax.security.auth.login.LoginContext.login(Unknown Source)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1092)
	... 21 more
Caused by: java.net.UnknownHostException: HOST
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(Unknown Source)
	at java.net.InetAddress.getAddressesFromNameService(Unknown Source)
	at java.net.InetAddress.getAllByName0(Unknown Source)
	at java.net.InetAddress.getAllByName(Unknown Source)
	at java.net.InetAddress.getAllByName(Unknown Source)
	at java.net.InetAddress.getByName(Unknown Source)
	at sun.security.krb5.internal.UDPClient.<init>(Unknown Source)
	at sun.security.krb5.internal.NetClient.getInstance(Unknown Source)
	at sun.security.krb5.KdcComm$KdcCommunication.run(Unknown Source)
	at sun.security.krb5.KdcComm$KdcCommunication.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.security.krb5.KdcComm.send(Unknown Source)
	at sun.security.krb5.KdcComm.sendIfPossible(Unknown Source)
	at sun.security.krb5.KdcComm.send(Unknown Source)
	at sun.security.krb5.KdcComm.send(Unknown Source)
	at sun.security.krb5.KrbAsReqBuilder.send(Unknown Source)
	at sun.security.krb5.KrbAsReqBuilder.action(Unknown Source)
	... 35 more

Does anyone have recommendations on how to do development and testing against a kerberized HBase cluster? We need some way to be able to run the client from a Windows box on our corporate network and have it work with Kerberos configured on our AWS cluster.

5 REPLIES 5

Re: Testing kerberized HBase cluster from a development machine

Super Guru

@Mike Thomsen

Do you have a VPC between your corporate and AWS? I am assuming your AWS can't be wide open? And that means the only way to connect from your laptop to your AWS is by setting up a VPC which you should already have. If you don't have a VPC then how do you connect from your laptop to your AWS instance?

You would have to download your hbase-site.xml, core-site.xml, hdfs-site.xml (not sure if this is required but why not) and use it for creating config in your java program. In these files change host to what you would use to connect to your AWS instance to your laptop.

Now if you don't have VPC and your current way to connect to AWS instance is using a key file then I am not sure you can do what you are trying to do.

Re: Testing kerberized HBase cluster from a development machine

Explorer

I should have been more specific. HOST is actually my laptop's hostname. It doesn't recognize my laptop when attempting to connect to the cluster, but if I move the same client I am using on my laptop to the cluster, it works just fine.

Re: Testing kerberized HBase cluster from a development machine

Super Guru

@Mike Thomsen

I understood that part. So let's say connectivity is not an issue which it appears from your log that it might not be. So, let me ask you this. The user you are connecting to HBase is not "hbase" and you are not using "hbase.keytab". Is that correct? One of the things you have in core-site.xml is the following.

  <property>
     <name>hadoop.proxyuser.hbase.hosts</name>
     <value>host1,host2</value> //does this include your laptop? Is this set to "*"?
   </property>
   <property>
     <name>hadoop.proxyuser.hbase.groups</name>
     <value>group1,group2</value>
   </property>

Also, you have same hbase-site.xml, core-site.xml on your laptop as the ones you have on your edge node where your program works?

check your hbase-site.xml and see if you have everything the way it expects.

Re: Testing kerberized HBase cluster from a development machine

You definitely don't need to set those configuration properties. The proxyuser properties are for allowing (or disallowing) intermediate services (e.g. HBase REST server) from accessing backend services (e.g. HBase) on your behalf. You do not need to set these values when you are accessing HBase directly.

Re: Testing kerberized HBase cluster from a development machine

"I should have been more specific. HOST is actually my laptop's hostname."

Java will normally find the KDC to communicate with to perform a login by the presence of a krb5.conf in expected locations. I don't know how (if at all, it seems) these are discovered when on Windows. I'd guess that it's just trying to talk to localhost because the JVM found no other configuration.

Thankfully, you can define the necessary information via system properties: the Kerberos realm to authenticate in and the IP address of the KDC. java.security.krb5.realm and java.security.krb5.kdc are the properties to set.

See https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/KerberosReq.html for more information on this. Also, beware that opening up a KDC to be accessible via the general internet is a scary decision. Please use caution when setting up the network rules.