@mbigelow   Any ideas?  ... View more

@mbigelow There are lots of logs like these in SM log: 2017-09-26 01:18:50,715 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x40b71007 connecting to ZooKeeper ensemble=inspire-dev-6:2181,inspire-dev-3:2181,inspire-dev-5:2181 2017-09-26 01:18:50,733 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x178b2a6f connecting to ZooKeeper ensemble=inspire-dev-6:2181,inspire-dev-3:2181,inspire-dev-5:2181 2017-09-26 01:19:50,735 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x4a38698e connecting to ZooKeeper ensemble=inspire-dev-6:2181,inspire-dev-3:2181,inspire-dev-5:2181 2017-09-26 01:19:50,754 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x60599b95 connecting to ZooKeeper ensemble=inspire-dev-6:2181,inspire-dev-3:2181,inspire-dev-5:2181 2017-09-26 01:20:50,771 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6f656e8a connecting to ZooKeeper ensemble=inspire-dev-6:2181,inspire-dev-3:2181,inspire-dev-5:2181 2017-09-26 01:20:50,792 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x22d3d22c connecting to ZooKeeper ensemble=inspire-dev-6:2181,inspire-dev-3:2181,inspire-dev-5:2181 2017-09-26 01:21:50,775 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x3b64eef7 connecting to ZooKeeper ensemble=inspire-dev-6:2181,inspire-dev-3:2181,inspire-dev-5:2181 2017-09-26 01:21:50,796 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x333cc0f1 connecting to ZooKeeper ensemble=inspire-dev-6:2181,inspire-dev-3:2181,inspire-dev-5:2181 And there are some WARNINGs : 2017-09-26 00:57:50,306 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0xf9c174f connecting to ZooKeeper ensemble=inspire-dev-6:2181,inspir e-dev-3:2181,inspire-dev-5:2181 2017-09-26 00:57:50,325 WARN com.cloudera.cmon.firehose.polling.CdhTask: (14 skipped) Exception in doWork for task: hbase_HBASE_SERVICE_STATE_TASK java.io.IOException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240) at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:412) at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:405) at org.apache.hadoop.hbase.client.ConnectionManager.getConnectionInternal(ConnectionManager.java:283) at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:116)   ... View more

hello: No your pointed kind of logs found, there are much number of this kind of logs, please check if it is useful:   2017-09-25 06:22:40,193 DEBUG org.apache.zookeeper.server.FinalRequestProcessor: sessionid:0x15eb7a1b1400001 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2017-09-25 06:22:41,739 DEBUG org.apache.zookeeper.server.quorum.CommitProcessor: Processing request:: sessionid:0x15eb7a1b1400002 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2017-09-25 06:22:41,739 DEBUG org.apache.zookeeper.server.FinalRequestProcessor: Processing request:: sessionid:0x15eb7a1b1400002 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2017-09-25 06:22:41,739 DEBUG org.apache.zookeeper.server.FinalRequestProcessor: sessionid:0x15eb7a1b1400002 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2017-09-25 06:22:42,193 DEBUG org.apache.zookeeper.server.quorum.CommitProcessor: Processing request:: sessionid:0x15eb7a1b1400001 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2017-09-25 06:22:42,193 DEBUG org.apache.zookeeper.server.FinalRequestProcessor: Processing request:: sessionid:0x15eb7a1b1400001 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2017-09-25 06:22:42,193 DEBUG org.apache.zookeeper.server.FinalRequestProcessor: sessionid:0x15eb7a1b1400001 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2017-09-25 06:22:42,394 DEBUG org.apache.zookeeper.server.quorum.CommitProcessor: Processing request:: sessionid:0x15e7e48142b2eb2 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2017-09-25 06:22:42,394 DEBUG org.apache.zookeeper.server.FinalRequestProcessor: Processing request:: sessionid:0x15e7e48142b2eb2 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2017-09-25 06:22:42,394 DEBUG org.apache.zookeeper.server.FinalRequestProcessor: sessionid:0x15e7e48142b2eb2 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Is it doing PING to zookeeper?   ... View more

  We need to figure out what SM is doing. Either go through the ZK logs to determine what node it is accessing. This may require you to increase the logging level. OK, increase it to DEBUG level? Now it is WARN level, and the log shows as: 2017-09-25 05:52:44,823 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Too many connections from /10.0.0.8 - max is 1000 2017-09-25 05:53:02,317 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Too many connections from /10.0.0.8 - max is 1000 2017-09-25 05:53:05,519 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Too many connections from /10.0.0.8 - max is 1000 2017-09-25 05:53:09,364 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Too many connections from /10.0.0.8 - max is 1000 2017-09-25 05:53:12,528 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Too many connections from /10.0.0.8 - max is 1000   I want to make sure that I got this right. The Canary test is disabled, since being disabled you have restarted the Service Monitor, and since restarting the connections from SM to ZK have climbed until hitting the maximum of 1000. Do I have that right? Yes, correct. I have restart it several times. ... View more

Now it's not the Canary problem, because I have closed the Canary test.  But the Zookeeper connections still be used up by SM. ... View more

My settings: ZooKeeper Canary Connection Timeout  : 10 seconds ZooKeeper Canary Session Timeout       : 30 seconds ZooKeeper Canary Operation Timeout    : 30 seconds   How many should I set to?     ... View more

Thanks for reply. Let me check. ... View more

Can we publish some Cloudera emploee? There are some questions that no one answers, if we can @   some professional emploee of Cloudera, that would be great. ... View more

Does my description unclear?  Can you give me some help? @mbigelow @csguna @saranvisa ... View more