2021-04-23 17:23:42,609 ccmp-hbase-ws DEBUG [http-nio-8111-exec-205] com.hbase.poc.controller.PocController - Inside the getTableInfo method with hbasepoc:alan_test and 1111-2021-01-05 2021-04-23 17:23:42,609 ccmp-hbase-ws DEBUG [http-nio-8111-exec-205] com.hbase.poc.dao.HbaseDAO - Ready to get data from table hbasepoc:alan_test, row key 1111-2021-01-05 2021-04-23 17:23:42,612 ccmp-hbase-ws DEBUG [http-nio-8111-exec-205] org.apache.hadoop.hbase.security.token.AuthenticationTokenSelector - No matching token found 2021-04-23 17:23:42,612 ccmp-hbase-ws DEBUG [http-nio-8111-exec-205] org.apache.hadoop.hbase.ipc.RpcConnection - RPC Server Kerberos principal name for service=ClientService is hbase/fepp-cdhdn-d2.mycompany.com@mycompany.com 2021-04-23 17:23:42,612 ccmp-hbase-ws DEBUG [http-nio-8111-exec-205] org.apache.hadoop.hbase.ipc.RpcConnection - Use KERBEROS authentication for service ClientService, sasl=true 2021-04-23 17:23:42,614 ccmp-hbase-ws DEBUG [Default-IPC-NioEventLoopGroup-1-2] org.apache.hadoop.hbase.security.AbstractHBaseSaslRpcClient - Creating SASL GSSAPI client. Server's Kerberos principal name is hbase/fepp-cdhdn-d2.mycompany.com@mycompany.com 2021-04-23 17:23:42,617 ccmp-hbase-ws DEBUG [Default-IPC-NioEventLoopGroup-1-2] org.apache.hadoop.security.UserGroupInformation - PrivilegedAction as:svc_fepd_deployment@mycompany.COM (auth:KERBEROS) from:org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.handlerAdded(NettyHBaseSaslRpcClientHandler.java:105) 2021-04-23 17:23:42,636 ccmp-hbase-ws DEBUG [Default-IPC-NioEventLoopGroup-1-2] org.apache.hadoop.security.UserGroupInformation - PrivilegedAction as:svc_fepd_deployment@mycompany.COM (auth:KERBEROS) from:org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.channelRead0(NettyHBaseSaslRpcClientHandler.java:138) 2021-04-23 17:23:42,637 ccmp-hbase-ws DEBUG [Default-IPC-NioEventLoopGroup-1-2] org.apache.hadoop.security.UserGroupInformation - PrivilegedAction as:svc_fepd_deployment@mycompany.COM (auth:KERBEROS) from:org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.channelRead0(NettyHBaseSaslRpcClientHandler.java:138) 2021-04-23 17:24:43,126 ccmp-hbase-ws DEBUG [http-nio-8111-exec-205] org.apache.hadoop.hbase.security.token.AuthenticationTokenSelector - No matching token found 2021-04-23 17:24:43,126 ccmp-hbase-ws DEBUG [http-nio-8111-exec-205] org.apache.hadoop.hbase.ipc.RpcConnection - RPC Server Kerberos principal name for service=ClientService is hbase/fepp-cdhdn-d5.mycompany.com@mycompany.com 2021-04-23 17:24:43,126 ccmp-hbase-ws DEBUG [http-nio-8111-exec-205] org.apache.hadoop.hbase.ipc.RpcConnection - Use KERBEROS authentication for service ClientService, sasl=true 2021-04-23 17:24:43,131 ccmp-hbase-ws DEBUG [Default-IPC-NioEventLoopGroup-1-3] org.apache.hadoop.hbase.security.AbstractHBaseSaslRpcClient - Creating SASL GSSAPI client. Server's Kerberos principal name is hbase/fepp-cdhdn-d5.mycompany.com@mycompany.com 2021-04-23 17:24:43,134 ccmp-hbase-ws DEBUG [Default-IPC-NioEventLoopGroup-1-3] org.apache.hadoop.security.UserGroupInformation - PrivilegedAction as:svc_fepd_deployment@mycompany.COM (auth:KERBEROS) from:org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.handlerAdded(NettyHBaseSaslRpcClientHandler.java:105) 2021-04-23 17:24:43,151 ccmp-hbase-ws DEBUG [Default-IPC-NioEventLoopGroup-1-3] org.apache.hadoop.security.UserGroupInformation - PrivilegedAction as:svc_fepd_deployment@mycompany.COM (auth:KERBEROS) from:org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.channelRead0(NettyHBaseSaslRpcClientHandler.java:138) 2021-04-23 17:24:43,153 ccmp-hbase-ws DEBUG [Default-IPC-NioEventLoopGroup-1-3] org.apache.hadoop.security.UserGroupInformation - PrivilegedAction as:svc_fepd_deployment@mycompany.COM (auth:KERBEROS) from:org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.channelRead0(NettyHBaseSaslRpcClientHandler.java:138) 2021-04-23 17:25:31,049 ccmp-hbase-ws DEBUG [Default-IPC-NioEventLoopGroup-1-2] org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler - Unknown callId: 36, skipping over this response of 62 bytes 2021-04-23 17:25:43,636 ccmp-hbase-ws INFO [ReadOnlyZKClient-fepp-cdhmn-d1.mycompany.com:2181,fepp-cdhmn-d2.mycompany.com:2181,fepp-cdhmn-d3.mycompany.com:2181@0x223e98e0] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=fepp-cdhmn-d1.mycompany.com:2181,fepp-cdhmn-d2.mycompany.com:2181,fepp-cdhmn-d3.mycompany.com:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$129/1042933680@f0dbec8 2021-04-23 17:25:43,647 ccmp-hbase-ws INFO [ReadOnlyZKClient-fepp-cdhmn-d1.mycompany.com:2181,fepp-cdhmn-d2.mycompany.com:2181,fepp-cdhmn-d3.mycompany.com:2181@0x223e98e0-SendThread(fepp-cdhmn-d1.mycompany.com:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server fepp-cdhmn-d1.mycompany.com/172.29.233.143:2181. Will not attempt to authenticate using SASL (unknown error) 2021-04-23 17:25:43,649 ccmp-hbase-ws INFO [ReadOnlyZKClient-fepp-cdhmn-d1.mycompany.com:2181,fepp-cdhmn-d2.mycompany.com:2181,fepp-cdhmn-d3.mycompany.com:2181@0x223e98e0-SendThread(fepp-cdhmn-d1.mycompany.com:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.29.162.9:39996, server: fepp-cdhmn-d1.mycompany.com/172.29.233.143:2181 2021-04-23 17:25:43,649 ccmp-hbase-ws DEBUG [ReadOnlyZKClient-fepp-cdhmn-d1.mycompany.com:2181,fepp-cdhmn-d2.mycompany.com:2181,fepp-cdhmn-d3.mycompany.com:2181@0x223e98e0-SendThread(fepp-cdhmn-d1.mycompany.com:2181)] org.apache.zookeeper.ClientCnxn - Session establishment request sent on fepp-cdhmn-d1.mycompany.com/172.29.233.143:2181 2021-04-23 17:25:43,666 ccmp-hbase-ws INFO [ReadOnlyZKClient-fepp-cdhmn-d1.mycompany.com:2181,fepp-cdhmn-d2.mycompany.com:2181,fepp-cdhmn-d3.mycompany.com:2181@0x223e98e0-SendThread(fepp-cdhmn-d1.mycompany.com:2181)] org.apache.zookeeper.ClientCnxn - Session establishment complete on server fepp-cdhmn-d1.mycompany.com/172.29.233.143:2181, sessionid = 0x178e597a63e6f91, negotiated timeout = 60000 2021-04-23 17:25:43,667 ccmp-hbase-ws DEBUG [ReadOnlyZKClient-fepp-cdhmn-d1.mycompany.com:2181,fepp-cdhmn-d2.mycompany.com:2181,fepp-cdhmn-d3.mycompany.com:2181@0x223e98e0-SendThread(fepp-cdhmn-d1.mycompany.com:2181)] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x178e597a63e6f91, packet:: clientPath:/hbase/meta-region-server serverPath:/hbase/meta-region-server finished:false header:: 1,4 replyHeader:: 1,579823280780,0 request:: '/hbase/meta-region-server,F response:: #ffffffff000146d61737465723a3136303030ffffffb7ffffff8affffffc6ffffff922b794dffffffb150425546a23a17666570702d636468646e2d64352e6665706f632e636f6d10ffffff947d18ffffff98ffffffdfffffffe1fffffff6ffffff8f2f100183,s{493921242528,579823225491,1583867566780,1619198033430,93,0,0,0,70,0,493921242528} 2021-04-23 17:26:03,667 ccmp-hbase-ws DEBUG [ReadOnlyZKClient-fepp-cdhmn-d1.mycompany.com:2181,fepp-cdhmn-d2.mycompany.com:2181,fepp-cdhmn-d3.mycompany.com:2181@0x223e98e0-SendThread(fepp-cdhmn-d1.mycompany.com:2181)] org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x178e597a63e6f91 after 0ms 2021-04-23 17:26:23,687 ccmp-hbase-ws DEBUG [ReadOnlyZKClient-fepp-cdhmn-d1.mycompany.com:2181,fepp-cdhmn-d2.mycompany.com:2181,fepp-cdhmn-d3.mycompany.com:2181@0x223e98e0-SendThread(fepp-cdhmn-d1.mycompany.com:2181)] org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x178e597a63e6f91 after 0ms 2021-04-23 17:26:28,390 ccmp-hbase-ws DEBUG [Default-IPC-NioEventLoopGroup-1-3] org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler - Unknown callId: 37, skipping over this response of 61 bytes 2021-04-23 17:26:28,406 ccmp-hbase-ws DEBUG [Default-IPC-NioEventLoopGroup-1-4] org.apache.hadoop.hbase.security.AbstractHBaseSaslRpcClient - Creating SASL GSSAPI client. Server's Kerberos principal name is hbase/fepp-cdhdn-d2.mycompany.com@mycompany.com 2021-04-23 17:26:28,410 ccmp-hbase-ws DEBUG [Default-IPC-NioEventLoopGroup-1-4] org.apache.hadoop.security.UserGroupInformation - PrivilegedAction as:svc_fepd_deployment@mycompany.COM (auth:KERBEROS) from:org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.handlerAdded(NettyHBaseSaslRpcClientHandler.java:105) 2021-04-23 17:26:28,426 ccmp-hbase-ws DEBUG [Default-IPC-NioEventLoopGroup-1-4] org.apache.hadoop.security.UserGroupInformation - PrivilegedAction as:svc_fepd_deployment@mycompany.COM (auth:KERBEROS) from:org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.channelRead0(NettyHBaseSaslRpcClientHandler.java:138) 2021-04-23 17:26:28,428 ccmp-hbase-ws DEBUG [Default-IPC-NioEventLoopGroup-1-4] org.apache.hadoop.security.UserGroupInformation - PrivilegedAction as:svc_fepd_deployment@mycompany.COM (auth:KERBEROS) from:org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.channelRead0(NettyHBaseSaslRpcClientHandler.java:138) 2021-04-23 17:26:43,668 ccmp-hbase-ws DEBUG [ReadOnlyZKClient-fepp-cdhmn-d1.mycompany.com:2181,fepp-cdhmn-d2.mycompany.com:2181,fepp-cdhmn-d3.mycompany.com:2181@0x223e98e0] org.apache.zookeeper.ZooKeeper - Closing session: 0x178e597a63e6f91 2021-04-23 17:26:43,668 ccmp-hbase-ws DEBUG [ReadOnlyZKClient-fepp-cdhmn-d1.mycompany.com:2181,fepp-cdhmn-d2.mycompany.com:2181,fepp-cdhmn-d3.mycompany.com:2181@0x223e98e0] org.apache.zookeeper.ClientCnxn - Closing client for session: 0x178e597a63e6f91 2021-04-23 17:26:43,683 ccmp-hbase-ws DEBUG [ReadOnlyZKClient-fepp-cdhmn-d1.mycompany.com:2181,fepp-cdhmn-d2.mycompany.com:2181,fepp-cdhmn-d3.mycompany.com:2181@0x223e98e0-SendThread(fepp-cdhmn-d1.mycompany.com:2181)] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x178e597a63e6f91, packet:: clientPath:null serverPath:null finished:false header:: 2,-11 replyHeader:: 2,579823280827,0 request:: null response:: null 2021-04-23 17:26:43,683 ccmp-hbase-ws DEBUG [ReadOnlyZKClient-fepp-cdhmn-d1.mycompany.com:2181,fepp-cdhmn-d2.mycompany.com:2181,fepp-cdhmn-d3.mycompany.com:2181@0x223e98e0] org.apache.zookeeper.ClientCnxn - Disconnecting client for session: 0x178e597a63e6f91 2021-04-23 17:26:43,683 ccmp-hbase-ws INFO [ReadOnlyZKClient-fepp-cdhmn-d1.mycompany.com:2181,fepp-cdhmn-d2.mycompany.com:2181,fepp-cdhmn-d3.mycompany.com:2181@0x223e98e0] org.apache.zookeeper.ZooKeeper - Session: 0x178e597a63e6f91 closed 2021-04-23 17:26:43,683 ccmp-hbase-ws INFO [ReadOnlyZKClient-fepp-cdhmn-d1.mycompany.com:2181,fepp-cdhmn-d2.mycompany.com:2181,fepp-cdhmn-d3.mycompany.com:2181@0x223e98e0-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down 2021-04-23 17:29:30,972 ccmp-hbase-ws ERROR [http-nio-8111-exec-205] com.hbase.poc.dao.HbaseDAO - getRow; Failed to get row 1111-2021-01-05 from table hbasepoc:alan_test org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=4, exceptions: Fri Apr 23 17:24:42 EDT 2021, RpcRetryingCaller{globalStartTime=1619213022609, pause=500, maxAttempts=4}, java.io.IOException: Call to fepp-cdhdn-d2.mycompany.com/172.29.233.141:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=36, waitTime=60009, rpcTimeout=60000 Fri Apr 23 17:27:28 EDT 2021, RpcRetryingCaller{globalStartTime=1619213022609, pause=500, maxAttempts=4}, java.io.IOException: Call to fepp-cdhdn-d2.mycompany.com/172.29.233.141:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=40, waitTime=60007, rpcTimeout=60000 Fri Apr 23 17:28:29 EDT 2021, RpcRetryingCaller{globalStartTime=1619213022609, pause=500, maxAttempts=4}, java.io.IOException: Call to fepp-cdhdn-d2.mycompany.com/172.29.233.141:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=43, waitTime=60007, rpcTimeout=60000 Fri Apr 23 17:29:30 EDT 2021, RpcRetryingCaller{globalStartTime=1619213022609, pause=500, maxAttempts=4}, java.io.IOException: Call to fepp-cdhdn-d2.mycompany.com/172.29.233.141:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=46, waitTime=60010, rpcTimeout=60000 at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:144) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:386) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:360) at com.hbase.poc.dao.HbaseDAO.getRow(HbaseDAO.java:41) at com.hbase.poc.dao.HbaseDAO.getRow(HbaseDAO.java:73) at com.hbase.poc.service.HbaseServiceImpl.getRow(HbaseServiceImpl.java:69) at com.hbase.poc.controller.PocController.getTableInfo(PocController.java:56) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115) at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59) at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to fepp-cdhdn-d2.mycompany.com/172.29.233.141:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=46, waitTime=60010, rpcTimeout=60000 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:185) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.setTimeout(Call.java:96) at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:199) at org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:663) at org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:738) at org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:466) ... 1 common frames omitted Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=46, waitTime=60010, rpcTimeout=60000 at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:200) ... 4 common frames omitted