Reply
New Contributor
Posts: 2
Registered: ‎04-23-2018

HBase not responding to Java calls after enabling Kerberos

have a web server (a Java application run without using Spark) which presents some data from a Cloudera cluster. The data are stored on HBase and the cluster is secured with Kerberos. When I try to perform a get, the server hangs without logging any error.

So far I've tried:

  1. Launching the webserver from command line after a kinit (the server is just for testing purpose, so log-in duration and complex procedures to start it are not an issue)
  2. The runAs approach described in here, both with and without the configuration file import from this answer.
  3. The CLASSPATH configuration approach described here
  4. Global authentication with UserGroupInformation.loginUserFromKeytab (with and without all the configurations from point 2 and 3)

I've executed all the gets from hbase shell after kiniting with the web server's user and they work in reasonable time (less than a second, while the last time I left the connection open the server didn't respond in over an hour), so it's not a performance or authorization issue. Inside the same web server, with every configuration listed, I'm able to perform other actions, like connecting to HBase and getting the table instances.

I've also checked the logs from Kerberos, HBase and my web server and none of them presents any error.

 

After some testing, I've verified that there is a user set right before the call to HBase's API and that said users is authorized to perform the operations I'm trying to execute (at leaste from the shell). Also, I've checked and no calls are made to HBase. So this is not an authentication problem, but something else. Did anybody have the same problem?

Posts: 1,657
Kudos: 321
Solutions: 260
Registered: ‎07-31-2013

Re: HBase not responding to Java calls after enabling Kerberos

Can you post a jstack output of your web server process? Also, are you
passing the right minimum set of configs (such as a hbase-site.xml copied
from a cluster's HBase gateway path - /etc/hbase/conf/) when building your
client?

You can follow the first/second option from here:
https://dzone.com/articles/how-to-take-thread-dumps-7-options
Highlighted
New Contributor
Posts: 2
Registered: ‎04-23-2018

Re: HBase not responding to Java calls after enabling Kerberos

After passing the path to the configuration files in the classpath I call:

Configuration hBaseConf = HBaseConfiguration.create();
hBaseConf.addResource("core-site.xml");
hBaseConf.addResource("hbase-site.xml");
hBaseConf.addResource("hdfs-site.xml");
hBaseConf.set("hadoop.security.authentication", "kerberos");
hBaseConf.set("hbase.security.authentication", "kerberos");

which should pass the configs you suggest. 

 

This is the jstack output:

2018-04-23 13:04:35
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.121-b13 mixed mode):

"Attach Listener" #30 daemon prio=9 os_prio=0 tid=0x00007fe4e8001000 nid=0x2453 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"pool-2-thread-4" #22 prio=5 os_prio=0 tid=0x00007fe4c4006800 nid=0x2396 waiting on condition [0x00007fe51ca47000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$1.run(RpcClientImpl.java:665)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:644)
	- locked <0x000000076f5fe2f0> (a org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:752)
	- locked <0x000000076f5fe2f0> (a org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:907)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:874)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1243)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:34070)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRowOrBefore(ProtobufUtil.java:1594)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1398)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1199)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1179)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1136)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:971)
	at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83)
	at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:79)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
	at org.apache.hadoop.hbase.client.HTable.get(HTable.java:930)
	at org.apache.hadoop.hbase.client.HTable.get(HTable.java:896)
	at server.QueryHandlers$Query.performQuery(QueryHandlers.java:65)
	at server.QueryHandlers$SingleQuery.handle(QueryHandlers.java:152)
	at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
	at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)
	at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)
	at sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)
	at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
	at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
	- <0x000000076dae22c8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"pool-2-thread-3" #21 prio=5 os_prio=0 tid=0x00007fe4c4004800 nid=0x2356 waiting on condition [0x00007fe51cb4a000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000076d9e8dd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
	- None

"pool-2-thread-2" #20 prio=5 os_prio=0 tid=0x00007fe4c4002800 nid=0x2320 waiting on condition [0x00007fe51cc4b000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000076d9e8dd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
	- None

"pool-2-thread-1" #19 prio=5 os_prio=0 tid=0x00007fe4c4001000 nid=0x231f waiting on condition [0x00007fe51cd4c000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000076d9e8dd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
	- None

"DestroyJavaVM" #18 prio=5 os_prio=0 tid=0x00007fe534011000 nid=0x2284 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"Thread-5" #17 prio=5 os_prio=0 tid=0x00007fe534bb1800 nid=0x22ab runnable [0x00007fe51ce4d000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x000000076d9db0d8> (a sun.nio.ch.Util$3)
	- locked <0x000000076d9db0c8> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000076d9dafb0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:352)
	at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
	- None

"server-timer" #16 daemon prio=5 os_prio=0 tid=0x00007fe534baf800 nid=0x22aa in Object.wait() [0x00007fe51cf4e000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x000000076d9dc808> (a java.util.TaskQueue)
	at java.util.TimerThread.mainLoop(Timer.java:552)
	- locked <0x000000076d9dc808> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:505)

   Locked ownable synchronizers:
	- None

"org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" #14 daemon prio=5 os_prio=0 tid=0x00007fe534b71800 nid=0x22a9 in Object.wait() [0x00007fe51d24f000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x000000076d53a708> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
	- locked <0x000000076d53a708> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
	at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3212)
	at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
	- None

"main-EventThread" #13 daemon prio=5 os_prio=0 tid=0x00007fe534adc800 nid=0x22a8 waiting on condition [0x00007fe51d350000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000771300740> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)

   Locked ownable synchronizers:
	- None

"main-SendThread(localhost:2181)" #12 daemon prio=5 os_prio=0 tid=0x00007fe534adf000 nid=0x22a7 runnable [0x00007fe51e70a000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000007715a5720> (a sun.nio.ch.Util$3)
	- locked <0x00000007715a5710> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000007715a5328> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

   Locked ownable synchronizers:
	- None

"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fe5340d0000 nid=0x2291 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fe5340c5000 nid=0x2290 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fe5340c3000 nid=0x228f waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fe5340c0000 nid=0x228e waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fe5340be800 nid=0x228d runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fe53408b800 nid=0x228b in Object.wait() [0x00007fe51f7f6000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x0000000771300c38> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
	- locked <0x0000000771300c38> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

   Locked ownable synchronizers:
	- None

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fe534087000 nid=0x228a in Object.wait() [0x00007fe51f8f7000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x0000000771300df0> (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
	- locked <0x0000000771300df0> (a java.lang.ref.Reference$Lock)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

   Locked ownable synchronizers:
	- None

"VM Thread" os_prio=0 tid=0x00007fe53407f800 nid=0x2289 runnable 

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fe534026800 nid=0x2285 runnable 

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fe534028000 nid=0x2286 runnable 

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fe53402a000 nid=0x2287 runnable 

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fe53402c000 nid=0x2288 runnable 

"VM Periodic Task Thread" os_prio=0 tid=0x00007fe5340d3800 nid=0x2292 waiting on condition 

JNI global references: 317

As you can see from the stack of pool-2-thread-4, it sleeps during the get execution (line 65 in my QueryHandlers.java calls mytable.Get)

Announcements