Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Failed to release ZooKeeper lock Exception in hiveserver2

Failed to release ZooKeeper lock Exception in hiveserver2

Explorer

hi ,

 

When we build MS sql cube from linked server connecting hiveserver2 with Cloudera's ODBC driver, it failed at most time.

I don't know if it has something to do with exception below:

 

2013-11-04 23:51:42,517 INFO  ql.Driver (PerfLogger.java:PerfLogEnd(127)) - </PERFLOG method=releaseLocks start=1383580302477 end=1383580302517 duration=40>
2013-11-04 23:51:42,517 INFO  ql.Driver (PerfLogger.java:PerfLogEnd(127)) - </PERFLOG method=Driver.run start=1383580302358 end=1383580302517 duration=159>
2013-11-04 23:52:40,485 ERROR ZooKeeperHiveLockManager (ZooKeeperHiveLockManager.java:unlockPrimitive(447)) - Failed to release ZooKeeper lock:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hive_zookeeper_namespace/<hiveDBName>/LOCK-SHARED-0000000000
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
        at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.unlockPrimitive(ZooKeeperHiveLockManager.java:435)
        at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.unlockWithRetry(ZooKeeperHiveLockManager.java:418)
        at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.unlock(ZooKeeperHiveLockManager.java:404)
        at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.releaseLocks(ZooKeeperHiveLockManager.java:251)
        at org.apache.hadoop.hive.ql.Driver.releaseLocks(Driver.java:885)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:981)
        at org.apache.hive.service.cli.operation.SQLOperation.run(SQLOperation.java:116)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:207)
        at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:154)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:191)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1193)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1178)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hive.service.cli.thrift.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:38)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2013-11-04 23:53:16,687 INFO  ql.Driver (PerfLogger.java:PerfLogBegin(100)) - <PERFLOG method=releaseLocks>
2013-11-04 23:53:16,690 INFO  ql.Driver (PerfLogger.java:PerfLogEnd(127)) - </PERFLOG method=releaseLocks start=1383580396687 end=1383580396690 duration=3>
2013-11-04 23:53:40,489 ERROR ZooKeeperHiveLockManager (ZooKeeperHiveLockManager.java:unlockPrimitive(447)) - Failed to release ZooKeeper lock:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hive_zookeeper_namespace/<hiveDBName>/LOCK-SHARED-0000000000
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
        at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.unlockPrimitive(ZooKeeperHiveLockManager.java:435)
        at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.unlockWithRetry(ZooKeeperHiveLockManager.java:418)
        at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.unlock(ZooKeeperHiveLockManager.java:404)
        at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.releaseLocks(ZooKeeperHiveLockManager.java:251)
        at org.apache.hadoop.hive.ql.Driver.releaseLocks(Driver.java:885)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:981)
        at org.apache.hive.service.cli.operation.SQLOperation.run(SQLOperation.java:116)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:207)

        at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:154)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:191)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1193)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1178)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hive.service.cli.thrift.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:38)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2013-11-04 23:54:40,540 ERROR ZooKeeperHiveLockManager (ZooKeeperHiveLockManager.java:unlockPrimitive(447)) - Failed to release ZooKeeper lock:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hive_zookeeper_namespace/<hiveDBName>/LOCK-SHARED-0000000000
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
        at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.unlockPrimitive(ZooKeeperHiveLockManager.java:435)
        at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.unlockWithRetry(ZooKeeperHiveLockManager.java:418)
        at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.unlock(ZooKeeperHiveLockManager.java:404)
        at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.releaseLocks(ZooKeeperHiveLockManager.java:251)
        at org.apache.hadoop.hive.ql.Driver.releaseLocks(Driver.java:885)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:981)
        at org.apache.hive.service.cli.operation.SQLOperation.run(SQLOperation.java:116)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:207)
        at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:154)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:191)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1193)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1178)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hive.service.cli.thrift.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:38)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2013-11-04 23:55:40,546 ERROR ZooKeeperHiveLockManager (ZooKeeperHiveLockManager.java:unlockPrimitive(447)) - Failed to release ZooKeeper lock:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hive_zookeeper_namespace/<hiveDBName>/LOCK-SHARED-0000000000
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
        at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.unlockPrimitive(ZooKeeperHiveLockManager.java:435)
        at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.unlockWithRetry(ZooKeeperHiveLockManager.java:418)
        at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.unlock(ZooKeeperHiveLockManager.java:404)
....

 and at the same time, we have logs below for zookeeper:

2013-11-04 23:59:32,790 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.162.153.51:51788 which had sessionid 0x142070ab9bf4265
2013-11-04 23:59:40,741 INFO org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x142070ab9bf425f type:delete cxid:0x1b zxid:0xd8ffc txntype:-1 reqpath:n/a Error Path:/hive_zookeeper_namespace/<hiveDBName> Error:KeeperErrorCode = NoNode for /hive_zookeeper_namespace/<hiveDBName>
2013-11-04 23:59:44,920 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.162.153.51:44987
2013-11-04 23:59:44,921 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.162.153.51:44987

 

CDH4.3 on CentOS6

 

Thanks.

1 REPLY 1

Re: Failed to release ZooKeeper lock Exception in hiveserver2

Explorer
I'm also seeing this too with CDH4.4, and it happens when I create multiple connections to Hive via JDBC. This post that I found is pretty much my scenario:

https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/_xSBdmwXSjI

Seeing there's been no response, I'm guessing 4.6 resolves it, but was hoping I could find a workaround in 4.4 before attempting an upgrade.
Don't have an account?
Coming from Hortonworks? Activate your account here