ZooKeeper Session expirations (e.g. "org.apache.zookeeper.KeeperException$SessionExpiredException:KeeperErrorCode=Session expired for/hbase-unsecure/rs/node03.test.com,16000,1453750627948") are a common issue that HBase can run into, although there are often a number of reasons which could cause this error.
Background: ZooKeeper clients need to maintain a "heartbeat" (a regular RPC to a ZooKeeper server) to maintain their session (credits to Apache ZooKeeper: http://zookeeper.apache.org/doc/r3.4.6/images/state_dia.jpg)
Without an active Session, clients cannot interact with ZooKeeper. In the HBase context, this means that HBase will continually try to connect to the server to put whatever state is necessary.
Some common reasons that this heartbeat fails:
- JVM garbage collections. Pauses in the HBase JVM can cause the heartbeat to not occur regularly. Look at JVM GC logs to identify this. HBase will also include messages in the service logs when it notices this happening.
- Host-level pauses. The node moving parts of the JVM out of memory and into swap (backed by disk) can cause pauses the entire JVM without any garbage collections. HBase will often notice this too, but report a pause without any garbage collections happening.
- Excess clients to ZooKeeper -- exceeding `maxClientCnxns` in zoo.cfg. See the ZooKeeper docs for a full description of this property, but the summary is that this property is used to rate-limit connections from one host. This is meant to prevent a denial of service attack, however, excessive load from a combination of HBase, MapReduce and other services can trigger this protection. In this case, clients are regularly trying to heartbeat to ZooKeeper, but the ZooKeeper server is actively *denying* the connection because it would exceed `maxClientCnxns` number of connections from the same host. You can use `netstat` to verify this is happening and/or check the ZooKeeper server log.
While removing the contents of HBase's root znode can be a temporary fix (especially with older versions which heavily rely on ZooKeeper for region assignments), this is often indicative of a much larger problem which will continue to occur in the future.
ZooKeeper Session expirations (e.g. "org.apache.zookeeper.KeeperException$SessionExpiredException:KeeperErrorCode=Session expired for/hbase-unsecure/rs/node03.test.com,16000,1453750627948") are a common issue that HBase can run into, although there are often a number of reasons which could cause this error.
Background: ZooKeeper clients need to maintain a "heartbeat" (a regular RPC to a ZooKeeper server) to maintain their session (credits to Apache ZooKeeper: http://zookeeper.apache.org/doc/r3.4.6/images/state_dia.jpg)
Without an active Session, clients cannot interact with ZooKeeper. In the HBase context, this means that HBase will continually try to connect to the server to put whatever state is necessary.
Some common reasons that this heartbeat fails:
- JVM garbage collections. Pauses in the HBase JVM can cause the heartbeat to not occur regularly. Look at JVM GC logs to identify this. HBase will also include messages in the service logs when it notices this happening.
- Host-level pauses. The node moving parts of the JVM out of memory and into swap (backed by disk) can cause pauses the entire JVM without any garbage collections. HBase will often notice this too, but report a pause without any garbage collections happening.
- Excess clients to ZooKeeper -- exceeding `maxClientCnxns` in zoo.cfg. See the ZooKeeper docs for a full description of this property, but the summary is that this property is used to rate-limit connections from one host. This is meant to prevent a denial of service attack, however, excessive load from a combination of HBase, MapReduce and other services can trigger this protection. In this case, clients are regularly trying to heartbeat to ZooKeeper, but the ZooKeeper server is actively *denying* the connection because it would exceed `maxClientCnxns` number of connections from the same host. You can use `netstat` to verify this is happening and/or check the ZooKeeper server log.
While removing the contents of HBase's root znode can be a temporary fix (especially with older versions which heavily rely on ZooKeeper for region assignments), this is often indicative of a much larger problem which will continue to occur in the future.
