Unable to create a znode for this server instance java.lang.Exception: Max znode creation wait time: 120s exhausted at org.apache.hive.service.server.HiveServer2.addServerInstanceToZooKeeper(HiveServer2.java:225) at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:417) at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:78) at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:654) at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:527) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) 2016-06-27 10:30:03,695 INFO [main]: server.HiveServer2 (HiveServer2.java:stop(371)) - Shutting down HiveServer2 2016-06-27 10:30:03,696 INFO [main]: thrift.ThriftCLIService (ThriftCLIService.java:stop(199)) - Thrift server has stopped 2016-06-27 10:30:03,696 INFO [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped. 2016-06-27 10:30:03,696 INFO [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped. 2016-06-27 10:30:03,696 INFO [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped. 2016-06-27 10:30:13,703 INFO [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:CLIService is stopped. 2016-06-27 10:30:13,703 INFO [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:HiveServer2 is stopped. 2016-06-27 10:30:13,713 INFO [main]: zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x3557c6d6327008e closed 2016-06-27 10:30:13,714 INFO [main]: server.HiveServer2 (HiveServer2.java:removeServerInstanceFromZooKeeper(338)) - Server instance removed from ZooKeeper. 2016-06-27 10:30:13,714 WARN [main]: server.HiveServer2 (HiveServer2.java:startHiveServer2(442)) - Error starting HiveServer2 on attempt 2, will retry in 60 seconds java.lang.Exception: Max znode creation wait time: 120s exhausted at org.apache.hive.service.server.HiveServer2.addServerInstanceToZooKeeper(HiveServer2.java:225) at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:417) at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:78) at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:654) at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:527) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) 2016-06-27 10:30:13,714 INFO [main-EventThread]: zookeeper.ClientCnxn (ClientCnxn.java:run(524)) - EventThread shut down 2016-06-27 10:30:45,032 INFO [HiveServer2-Handler-Pool: Thread-45]: session.HiveSessionImpl (HiveSessionImpl.java:acquire(304)) - We are setting the hadoop caller context to 6496180e-e8df-47c6-9bbb-c53f54059d9e for thread HiveServer2-Handler-Pool: Thread-45 2016-06-27 10:30:45,033 INFO [HiveServer2-Handler-Pool: Thread-45]: operation.Operation (HiveCommandOperation.java:setupSessionIO(69)) - Putting temp output to file /tmp/hive/6496180e-e8df-47c6-9bbb-c53f54059d9e5129184338741826418.pipeout 2016-06-27 10:30:45,034 WARN [HiveServer2-Handler-Pool: Thread-45]: operation.Operation (Operation.java:createOperationLog(206)) - Unable to create operation log file: /tmp/hive/operation_logs/6496180e-e8df-47c6-9bbb-c53f54059d9e/da377682-6244-4c72-977c-70e0803e61c0 java.io.IOException: No such file or directory at java.io.UnixFileSystem.createFileExclusively(Native Method) at java.io.File.createNewFile(File.java:1012) at org.apache.hive.service.cli.operation.Operation.createOperationLog(Operation.java:195) at org.apache.hive.service.cli.operation.Operation.beforeRun(Operation.java:237) at org.apache.hive.service.cli.operation.Operation.run(Operation.java:255) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:419) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:406) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) at com.sun.proxy.$Proxy20.executeStatementAsync(Unknown Source) at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:274) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-06-27 10:30:45,035 INFO [HiveServer2-Handler-Pool: Thread-45]: SessionState (SessionState.java:printInfo(923)) - Added [/usr/hdp/2.4.0.0-169/hbase/lib/hbase-common.jar] to class path 2016-06-27 10:30:45,035 INFO [HiveServer2-Handler-Pool: Thread-45]: SessionState (SessionState.java:printInfo(923)) - Added resources: [/usr/hdp/2.4.0.0-169/hbase/lib/hbase-common.jar] 2016-06-27 10:30:45,035 INFO [HiveServer2-Handler-Pool: Thread-45]: session.HiveSessionImpl (HiveSessionImpl.java:release(318)) - We are resetting the hadoop caller context for thread HiveServer2-Handler-Pool: Thread-45 2016-06-27 10:30:45,036 INFO [HiveServer2-Handler-Pool: Thread-45]: thrift.ThriftCLIService (ThriftCLIService.java:deleteContext(118)) - Session disconnected without closing properly, close it now 2016-06-27 10:30:45,036 INFO [HiveServer2-Handler-Pool: Thread-45]: session.HiveSessionImpl (HiveSessionImpl.java:acquire(304)) - We are setting the hadoop caller context to 6496180e-e8df-47c6-9bbb-c53f54059d9e for thread HiveServer2-Handler-Pool: Thread-45 2016-06-27 10:30:45,036 INFO [HiveServer2-Handler-Pool: Thread-45]: session.HiveSessionImpl (HiveSessionImpl.java:acquire(304)) - We are setting the hadoop caller context to 6496180e-e8df-47c6-9bbb-c53f54059d9e for thread HiveServer2-Handler-Pool: Thread-45 2016-06-27 10:30:45,036 ERROR [HiveServer2-Handler-Pool: Thread-45]: session.HiveSessionImpl (HiveSessionImpl.java:cleanupSessionLogDir(622)) - Failed to cleanup session log dir: SessionHandle [6496180e-e8df-47c6-9bbb-c53f54059d9e] java.io.FileNotFoundException: File does not exist: /tmp/hive/operation_logs/6496180e-e8df-47c6-9bbb-c53f54059d9e at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2275) at org.apache.hive.service.cli.session.HiveSessionImpl.cleanupSessionLogDir(HiveSessionImpl.java:620) at org.apache.hive.service.cli.session.HiveSessionImpl.close(HiveSessionImpl.java:592) at org.apache.hive.service.cli.session.HiveSessionImplwithUGI.close(HiveSessionImplwithUGI.java:108) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) at com.sun.proxy.$Proxy20.close(Unknown Source) at org.apache.hive.service.cli.session.SessionManager.closeSession(SessionManager.java:292) at org.apache.hive.service.cli.CLIService.closeSession(CLIService.java:237) at org.apache.hive.service.cli.thrift.ThriftCLIService$1.deleteContext(ThriftCLIService.java:120) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:300) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-06-27 10:30:45,053 INFO [HiveServer2-Handler-Pool: Thread-45]: session.HiveSessionImpl (HiveSessionImpl.java:release(318)) - We are resetting the hadoop caller context for thread HiveServer2-Handler-Pool: Thread-45 2016-06-27 10:30:45,057 INFO [HiveServer2-Handler-Pool: Thread-45]: service.CompositeService (SessionManager.java:closeSession(300)) - This instance of HiveServer2 has been removed from the list of server instances available for dynamic service discovery. The last client session has ended - will shutdown now. 2016-06-27 10:30:45,058 INFO [Thread-25]: server.HiveServer2 (HiveServer2.java:stop(371)) - Shutting down HiveServer2 2016-06-27 10:30:45,058 INFO [Thread-25]: server.HiveServer2 (HiveServer2.java:removeServerInstanceFromZooKeeper(338)) - Server instance removed from ZooKeeper. 2016-06-27 10:31:13,714 INFO [main]: server.HiveServer2 (HiveServer2.java:startHiveServer2(405)) - Starting HiveServer2 2016-06-27 10:31:14,009 INFO [main]: hive.metastore (HiveMetaStoreClient.java:isCompatibleWith(296)) - Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook to org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl 2016-06-27 10:31:14,010 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(747)) - 0: Shutting down the object store... 2016-06-27 10:31:14,011 INFO [main]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(372)) - ugi=hive ip=unknown-ip-addr cmd=Shutting down the object store... 2016-06-27 10:31:14,011 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(747)) - 0: Metastore shutdown complete. 2016-06-27 10:31:14,012 INFO [main]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(372)) - ugi=hive ip=unknown-ip-addr cmd=Metastore shutdown complete. 2016-06-27 10:31:14,035 INFO [main]: session.SessionState (SessionState.java:createPath(613)) - Created local directory: /tmp/a1490877-0073-4edd-8729-048b7159f57b_resources 2016-06-27 10:31:14,040 INFO [main]: session.SessionState (SessionState.java:createPath(613)) - Created HDFS directory: /tmp/hive/hive/a1490877-0073-4edd-8729-048b7159f57b 2016-06-27 10:31:14,046 INFO [main]: session.SessionState (SessionState.java:createPath(613)) - Created local directory: /tmp/hive/a1490877-0073-4edd-8729-048b7159f57b 2016-06-27 10:31:14,056 INFO [main]: session.SessionState (SessionState.java:createPath(613)) - Created HDFS directory: /tmp/hive/hive/a1490877-0073-4edd-8729-048b7159f57b/_tmp_space.db 2016-06-27 10:31:14,070 INFO [main]: sqlstd.SQLStdHiveAccessController (SQLStdHiveAccessController.java:(95)) - Created SQLStdHiveAccessController for session context : HiveAuthzSessionContext [sessionString=a1490877-0073-4edd-8729-048b7159f57b, clientType=HIVESERVER2] 2016-06-27 10:31:14,071 INFO [main]: hive.metastore (HiveMetaStoreClient.java:isCompatibleWith(296)) - Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook 2016-06-27 10:31:14,071 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(747)) - 0: Shutting down the object store... 2016-06-27 10:31:14,072 INFO [main]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(372)) - ugi=hive ip=unknown-ip-addr cmd=Shutting down the object store... 2016-06-27 10:31:14,072 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(747)) - 0: Metastore shutdown complete. 2016-06-27 10:31:14,073 INFO [main]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(372)) - ugi=hive ip=unknown-ip-addr cmd=Metastore shutdown complete. 2016-06-27 10:31:14,074 INFO [main]: service.CompositeService (SessionManager.java:initOperationLogRootDir(135)) - Operation log root directory is created: /tmp/hive/operation_logs 2016-06-27 10:31:14,075 INFO [main]: service.CompositeService (SessionManager.java:createBackgroundOperationPool(90)) - HiveServer2: Background operation thread pool size: 100 2016-06-27 10:31:14,075 INFO [main]: service.CompositeService (SessionManager.java:createBackgroundOperationPool(92)) - HiveServer2: Background operation thread wait queue size: 100 2016-06-27 10:31:14,075 INFO [main]: service.CompositeService (SessionManager.java:createBackgroundOperationPool(95)) - HiveServer2: Background operation thread keepalive time: 10 seconds 2016-06-27 10:31:14,076 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:OperationManager is inited. 2016-06-27 10:31:14,076 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:SessionManager is inited. 2016-06-27 10:31:14,076 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:CLIService is inited. 2016-06-27 10:31:14,080 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:ThriftBinaryCLIService is inited. 2016-06-27 10:31:14,080 INFO [main]: service.AbstractService (AbstractService.java:init(89)) - Service:HiveServer2 is inited. 2016-06-27 10:31:14,080 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:OperationManager is started. 2016-06-27 10:31:14,081 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:SessionManager is started. 2016-06-27 10:31:14,081 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:CLIService is started. 2016-06-27 10:31:14,082 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(747)) - 0: get_databases: default 2016-06-27 10:31:14,083 INFO [main]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(372)) - ugi=hive ip=unknown-ip-addr cmd=get_databases: default 2016-06-27 10:31:14,083 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:newRawStore(590)) - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 2016-06-27 10:31:14,092 INFO [main]: metastore.ObjectStore (ObjectStore.java:initialize(294)) - ObjectStore, initialize called 2016-06-27 10:31:14,118 INFO [main]: metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:(140)) - Using direct SQL, underlying DB is MYSQL 2016-06-27 10:31:14,118 INFO [main]: metastore.ObjectStore (ObjectStore.java:setConf(277)) - Initialized ObjectStore 2016-06-27 10:31:14,135 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(747)) - 0: Shutting down the object store... 2016-06-27 10:31:14,136 INFO [main]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(372)) - ugi=hive ip=unknown-ip-addr cmd=Shutting down the object store... 2016-06-27 10:31:14,136 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(747)) - 0: Metastore shutdown complete. 2016-06-27 10:31:14,137 INFO [main]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(372)) - ugi=hive ip=unknown-ip-addr cmd=Metastore shutdown complete. 2016-06-27 10:31:14,137 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:ThriftBinaryCLIService is started. 2016-06-27 10:31:14,139 INFO [main]: service.AbstractService (AbstractService.java:start(104)) - Service:HiveServer2 is started. 2016-06-27 10:31:14,142 INFO [main]: imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:start(224)) - Starting 2016-06-27 10:31:14,145 INFO [main]: zookeeper.ZooKeeper (ZooKeeper.java:(438)) - Initiating client connection, connectString=zlake012.xyzsolutions.local:2181,zlake013.xyzsolutions.local:2181,zlake011.xyzsolutions.local:2181 sessionTimeout=1200000 watcher=org.apache.curator.ConnectionState@c1e14f 2016-06-27 10:31:14,150 INFO [Thread-29]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(98)) - Starting ThriftBinaryCLIService on port 10000 with 5...500 worker threads 2016-06-27 10:31:14,151 INFO [main-SendThread(zlake013.xyzsolutions.local:2181)]: zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server zlake013.xyzsolutions.local/172.16.3.197:2181. Will not attempt to authenticate using SASL (unknown error) 2016-06-27 10:31:14,152 INFO [main-SendThread(zlake013.xyzsolutions.local:2181)]: zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to zlake013.xyzsolutions.local/172.16.3.197:2181, initiating session 2016-06-27 10:31:14,156 INFO [main-SendThread(zlake013.xyzsolutions.local:2181)]: zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1279)) - Session establishment complete on server zlake013.xyzsolutions.local/172.16.3.197:2181, sessionid = 0x3557c6d6327008f, negotiated timeout = 40000 2016-06-27 10:31:14,157 INFO [main-EventThread]: state.ConnectionStateManager (ConnectionStateManager.java:postState(228)) - State change: CONNECTED