Hi,
I would like to ask for a little help on an error I have on Hbase 1.0.0.
I have :
- CDH 5.4 on Kerberized Cluster: 4 nodes
(Hdfs, YARN, Hive and Sentry services are working without issues)
- HBase 1.0.0-cdh5.4.4
(3 RegionServers, 1 Master Active, 1 REST Server, 1 Thrift Server)
- Zookeeper
(3 nodes)
When I connect with Java API to an Hbase table (created via shell), if i try to execute any method apart getName(that instead works) I get this exception :
Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=68248: row 'tb_name,,00000000000000' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=myhostname,60020,1448983052715, seqNum=0
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to hostname/10.180.113.56:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to hostname/10.180.113.56:60020 is closing. Call id=9, waitTime=2
at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1233)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1204)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216)
atorg.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300)
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:31889)
at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:349)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:193)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:332)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:306)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
... 4 more
Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to hostname/10.180.113.56:60020 is closing. Call id=9, waitTime=2
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.cleanupCalls(RpcClientImpl.java:1033)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:840)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:568)
I already tried to increment the RPC timeout but didnt work.
The table contains only one row with one columnFamily.
Please let me know if someone have already seen this problem and can help.
Accessing hbase from the shell seem to work properly,I can scan the table, cerate new table and get the row using GET.
Thanks for your help
Ps
I want to add some more infos :
result from hbase hbck:
15/12/07 15:10:54 ERROR master.TableLockManager: Unexpected ZooKeeper error when listing children
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/table-lock
at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:296)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenNoWatch(ZKUtil.java:575)
at org.apache.hadoop.hbase.master.TableLockManager$ZKTableLockManager.getTableNames(TableLockManager.java:392)
at org.apache.hadoop.hbase.master.Table
LockManager$ZKTableLockManager.visitAllLocks(TableLockManager.java:379)
at org.apache.hadoop.hbase.util.hbck.TableLockChecker.checkTableLocks(TableLockChecker.java:76)
at org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixTableLocks(HBaseFsck.java:3026)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:629)
at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4440)
at org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:4243)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:4231)
result from hbase zkcli:
[ERROR] Terminal initialization failed; falling back to unsupported
java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected
at jline.TerminalFactory.create(TerminalFactory.java:101)
at jline.TerminalFactory.get(TerminalFactory.java:159)
at jline.console.ConsoleReader.<init>(ConsoleReader.java:227)
at jline.console.ConsoleReader.<init>(ConsoleReader.java:219)
at jline.console.ConsoleReader.<init>(ConsoleReader.java:207)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImp
l.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:311)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperMainServer.main(ZooKeeperMainServer.java:136)
JLine support is disabled
15/12/07 15:13:35 INFO zookeeper.Login: TGT valid starting at: Mon Dec 07 15:13:12 CET 2015
15/12/07 15:13:35 INFO zookeeper.Login: TGT expires: Tue Dec 08 01:13:15 CET 2015
15/12/07 15:13:35 INFO zookeeper.Login: TGT refresh sleeping until: Mon Dec 07 23:35:04 CET 2015
15/12/07 15:13:35 INFO zookeeper.ClientCnxn: Opening socket connection to server lmldrl38cds001.generali.it/10.180.113.55:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
15/12/07 15:13:35 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.180.113.54:53649, server: lmldrl38cds001.generali.it/10.180.113.55:2181
15/12/07 15:13:35 INFO zookeeper.ClientCnxn: Session establishment complete on server hostname/10.180.113.55:2181, sessionid = 0x3516cba6a2b174f, negotiated timeout = 30000
Hide full text
3:06:44.929 PM ERROR org.apache.zookeeper.server.auth.SaslServerCallbackHandler
Moreover from the log accesible from Cloudera manager i can see this line related to Kerberos Authentication:
3:06:44.929 PM ERROR org.apache.zookeeper.server.auth.SaslServerCallbackHandler Failed to set name based on Kerberos authentication rules.
Hopes this more infos can help to identify the issue
