Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Zookeeper connections are taken up by service monitor

Highlighted

Re: Zookeeper connections are taken up by service monitor

Champion
That appears to be between the ZK servers. Anything useful in the SM logs?

Re: Zookeeper connections are taken up by service monitor

Explorer

@mbigelow

There are lots of logs like these in SM log:

2017-09-26 01:18:50,715 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x40b71007 connecting to ZooKeeper ensemble=inspire-dev-6:2181,inspire-dev-3:2181,inspire-dev-5:2181
2017-09-26 01:18:50,733 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x178b2a6f connecting to ZooKeeper ensemble=inspire-dev-6:2181,inspire-dev-3:2181,inspire-dev-5:2181
2017-09-26 01:19:50,735 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x4a38698e connecting to ZooKeeper ensemble=inspire-dev-6:2181,inspire-dev-3:2181,inspire-dev-5:2181
2017-09-26 01:19:50,754 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x60599b95 connecting to ZooKeeper ensemble=inspire-dev-6:2181,inspire-dev-3:2181,inspire-dev-5:2181
2017-09-26 01:20:50,771 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6f656e8a connecting to ZooKeeper ensemble=inspire-dev-6:2181,inspire-dev-3:2181,inspire-dev-5:2181
2017-09-26 01:20:50,792 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x22d3d22c connecting to ZooKeeper ensemble=inspire-dev-6:2181,inspire-dev-3:2181,inspire-dev-5:2181
2017-09-26 01:21:50,775 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x3b64eef7 connecting to ZooKeeper ensemble=inspire-dev-6:2181,inspire-dev-3:2181,inspire-dev-5:2181
2017-09-26 01:21:50,796 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x333cc0f1 connecting to ZooKeeper ensemble=inspire-dev-6:2181,inspire-dev-3:2181,inspire-dev-5:2181

And there are some WARNINGs :

2017-09-26 00:57:50,306 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0xf9c174f connecting to ZooKeeper ensemble=inspire-dev-6:2181,inspir
e-dev-3:2181,inspire-dev-5:2181
2017-09-26 00:57:50,325 WARN com.cloudera.cmon.firehose.polling.CdhTask: (14 skipped) Exception in doWork for task: hbase_HBASE_SERVICE_STATE_TASK
java.io.IOException: java.lang.reflect.InvocationTargetException
        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
        at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:412)
        at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:405)
        at org.apache.hadoop.hbase.client.ConnectionManager.getConnectionInternal(ConnectionManager.java:283)
        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:116)

 

Re: Zookeeper connections are taken up by service monitor

Explorer

@mbigelow  

Any ideas?