https://community.cloudera.com/t5/Support-Questions/Hbase-region-servers-not-able-to-connect-to-master-after/m-p/198708#M160756 <P>Hi <A rel="user" href="https://community.cloudera.com/users/223/jelser.html" nodeid="223">@Josh Elser</A>, this is what i could see in the logs in debug mode. I am seeing a ipc timeout exception.</P><P>rpc timeout is set around 15 mins. Any other parameter that i should look in?</P><P>2017-04-18 10:41:51,628 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] ipc.AbstractRpcClient: Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e4158de, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null 2017-04-18 10:41:51,686 INFO [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.HRegionServer: ClusterId : a6d99271-51ab-41ed-a2d5-93049dedd72d 2017-04-18 10:41:51,693 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] procedure.RegionServerProcedureManagerHost: Procedure flush-table-proc is initializing 2017-04-18 10:41:51,760 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] zookeeper.RecoverableZooKeeper: Node /hbase-unsecure/flush-table-proc/acquired already exists 2017-04-18 10:41:51,782 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] procedure.RegionServerProcedureManagerHost: Procedure flush-table-proc is initialized 2017-04-18 10:41:51,782 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] procedure.RegionServerProcedureManagerHost: Procedure online-snapshot is initializing 2017-04-18 10:41:51,786 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] zookeeper.RecoverableZooKeeper: Node /hbase-unsecure/online-snapshot/acquired already exists 2017-04-18 10:41:51,789 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] procedure.RegionServerProcedureManagerHost: Procedure online-snapshot is initialized 2017-04-18 10:41:51,793 WARN [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.lowerLimit is deprecated. Instead use hbase.regionserver.global.memstore.size.lower.limit 2017-04-18 10:41:51,795 INFO [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.MemStoreFlusher: globalMemStoreLimit=1.6 G, globalMemStoreLimitLowMark=1003.6 M, maxHeap=3.9 G 2017-04-18 10:41:51,800 INFO [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.HRegionServer: CompactionChecker runs every 10sec 2017-04-18 10:41:51,818 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] ipc.AbstractRpcClient: Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ea71264, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=abcd013.abcdsolutions.local/172.16.3.197:0 2017-04-18 10:41:51,823 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.ShutdownHook: Installed shutdown hook thread: Shutdownhook:regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020 2017-04-18 10:41:51,870 INFO [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.RegionServerCoprocessorHost: System coprocessor loading is enabled 2017-04-18 10:41:51,870 INFO [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.RegionServerCoprocessorHost: Table coprocessor loading is enabled 2017-04-18 10:41:51,877 INFO [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.HRegionServer: reportForDuty to master=abcd011.abcdsolutions.local,16000,1492492302253 with port=16020, startcode=1492492307927 2017-04-18 10:41:52,101 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] ipc.AbstractRpcClient: Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2017-04-18 10:41:52,154 DEBUG [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] ipc.AbstractRpcClient: Connecting to abcd011.abcdsolutions.local/172.16.3.195:16000 2017-04-18 10:42:02,024 WARN [regionserver/abcd013.abcdsolutions.local/172.16.3.197:16020] regionserver.HRegionServer: error telling master we are up com.google.protobuf.ServiceException: java.io.IOException: Call to abcd011.abcdsolutions.local/172.16.3.195:16000 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=0, waitTime=10001, operationTimeout=10000 expired. at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:10859) at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2406) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:936)</P> Tue, 18 Apr 2017 18:43:44 GMT arunpoy 2017-04-18T18:43:44Z