Cloudera Community
Announcements
Now Live: Explore expert insights and technical deep dives on the new Cloudera Community Blogs — Read the Announcement
mike_bronson7
user rank
Guru

matrics collector is up but metrics dashboard not ...

by Guru in Support Questions
‎06-15-2018 02:51 PM
‎06-15-2018 02:51 PM
hi all we have ambari cluster , and matrics collector is up but metrics dashboard not display usage ( no data availble ) as the following ... View more

Re: zookeper - Too many connections

by Guru in Support Questions
‎06-12-2018 06:52 PM
‎06-12-2018 06:52 PM
in the first zoo log we can see also the following: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2018-06-12 18:35:01,856 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2018-06-12 18:35:01,857 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2018-06-12 18:35:01,857 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2018-06-12 18:35:01,857 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2018-06-12 18:35:01,857 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2018-06-12 18:35:01,857 - ERROR [CommitProcessor:1:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1082) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:391) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) ... View more

Re: zookeper - Too many connections

by Guru in Support Questions
‎06-12-2018 06:50 PM
‎06-12-2018 06:50 PM
see also the following details ( sent much less then Received ) , is it can tell us something about the problem ? this happend only on the first zoo server echo stat | nc 10414.42.169 2181 Latency min/avg/max: 0/10/2727 Received: 600879 Sent: 103803 Connections: 30 Outstanding: 546 Zxid: 0x3e000048c3 Mode: follower Node count: 43296 ... View more

Re: zookeper - Too many connections

by Guru in Support Questions
‎06-12-2018 06:07 PM
‎06-12-2018 06:07 PM
I can kill all CLOSE-WAIT by - lsof -i :2181 | grep CLOSE_WAIT| awk '{print $2}' |uniq| xargs kill , but this isnt solution , why zoo not close them ? ... View more

Re: zookeper - Too many connections

by Guru in Support Questions
‎06-12-2018 05:57 PM
‎06-12-2018 05:57 PM
Geoffrey , this cluster is diff cluster , not the clkuster with the namenode isshue , ... View more

Re: zookeper - Too many connections

by Guru in Support Questions
‎06-12-2018 10:42 AM
‎06-12-2018 10:42 AM
we kill the CLOSE-WAIT and restart the zoo , but still zoo connection are more then max , what we can do with this ? ... View more

Re: zookeper - Too many connections

by Guru in Support Questions
‎06-12-2018 10:09 AM
‎06-12-2018 10:09 AM
do you mean to kill ( TIME-WAIT , CLOSE-WAIT ) ... View more

Re: zookeper - Too many connections

by Guru in Support Questions
‎06-12-2018 07:55 AM
‎06-12-2018 07:55 AM
on master01 we have 172 conections , and master02/03 we have ~50 ... View more

zookeper - Too many connections

by Guru in Support Questions
‎06-12-2018 06:29 AM
‎06-12-2018 06:29 AM
in the zoo logs we get the following warning : we also get disconnecting issues from zookeper , is it possible to increase the "60" value? what should be the ressolution for this ? Too many connections from /10.54.23.11 - max is 60 from zoo log: 2018-06-12 06:31:56,901 - ERROR [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$1@44] - Thread Thread[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181,5,main] died java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:934) at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:745 2018-06-12 06:18:16,687 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for client /10.164.46.204:52745 which had sessionid 0x163f299c1e20003 2018-06-12 06:18:16,687 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x163f299c1e20003 2018-06-12 06:18:16,688 - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@617] - Established session 0x163f299c1e20003 with negotiated timeout 40000 for client /10.164.46.204:52835 2018-06-12 06:18:16,878 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.54.23.11 - max is 60 2018-06-12 06:18:17,550 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.164.46.170:42523 2018-06-12 06:18:17,550 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x163f299c1e20004 at /10.54.23.12:42523 2018-06-12 06:18:17,550 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for client /10.164.46.170:42495 which had sessionid 0x163f299c1e20004 2018-06-12 06:18:17,551 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x163f299c1e20004 2018-06-12 06:18:17,552 - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@617] - Established session 0x163f299c1e20004 with negotiated timeout 40000 for client /10.164.46.170:42523 2018-06-12 06:18:18,296 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.54.23.11 - max is 60 2018-06-12 06:18:19,695 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.54.23.11 - max is 60 2018-06-12 06:18:21,042 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.54.23.11 - max is 60 2018-06-12 06:18:21,761 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.54.23.11 - max is 60 2018-06-12 06:18:22,408 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.54.23.11 - max is 60 2018-06-12 06:18:23,561 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.54.23.11 - max is 60 2018-06-12 06:18:24,192 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.54.23.11 - max is 60 ^C [root@master01 zookeeper]# host 10.54.23.11 169.46.164.10.in-addr.arpa domain name pointer master01.sys76.com. ... View more

Re: How to clear /clean-up pg_xlog directory? on a...

by Guru in Support Questions
‎06-11-2018 05:41 AM
‎06-11-2018 05:41 AM
what variable represented the - "enabled replication for this Postgres instance" ? ... View more
Contact Me
Online
Offline
Last Visited
‎08-27-2024 09:17 AM
Community Statistics
Member Since ‎08-08-2017 09:40 AM
Last Visited ‎08-27-2024 09:17 AM
Posts 1,652
Kudos received 29