About mike_bronson7

mike_bronson7 · ‎06-12-2018

Geoffrey , this cluster is diff cluster , not the clkuster with the namenode isshue ,

mike_bronson7 · ‎06-12-2018

we kill the CLOSE-WAIT and restart the zoo , but still zoo connection are more then max , what we can do with this ?

mike_bronson7 · ‎06-12-2018

do you mean to kill ( TIME-WAIT , CLOSE-WAIT )

mike_bronson7 · ‎06-12-2018

on master01 we have 172 conections , and master02/03 we have ~50

mike_bronson7 · ‎06-12-2018

in the zoo logs we get the following warning : we also get disconnecting issues from zookeper , is it possible to increase the "60" value? what should be the ressolution for this ? Too many connections from /10.54.23.11 - max is 60 from zoo log: 2018-06-12 06:31:56,901 - ERROR [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$1@44] - Thread Thread[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181,5,main] died java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:934) at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:745 2018-06-12 06:18:16,687 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for client /10.164.46.204:52745 which had sessionid 0x163f299c1e20003 2018-06-12 06:18:16,687 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x163f299c1e20003 2018-06-12 06:18:16,688 - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@617] - Established session 0x163f299c1e20003 with negotiated timeout 40000 for client /10.164.46.204:52835 2018-06-12 06:18:16,878 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.54.23.11 - max is 60 2018-06-12 06:18:17,550 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.164.46.170:42523 2018-06-12 06:18:17,550 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x163f299c1e20004 at /10.54.23.12:42523 2018-06-12 06:18:17,550 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for client /10.164.46.170:42495 which had sessionid 0x163f299c1e20004 2018-06-12 06:18:17,551 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x163f299c1e20004 2018-06-12 06:18:17,552 - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@617] - Established session 0x163f299c1e20004 with negotiated timeout 40000 for client /10.164.46.170:42523 2018-06-12 06:18:18,296 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.54.23.11 - max is 60 2018-06-12 06:18:19,695 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.54.23.11 - max is 60 2018-06-12 06:18:21,042 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.54.23.11 - max is 60 2018-06-12 06:18:21,761 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.54.23.11 - max is 60 2018-06-12 06:18:22,408 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.54.23.11 - max is 60 2018-06-12 06:18:23,561 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.54.23.11 - max is 60 2018-06-12 06:18:24,192 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.54.23.11 - max is 60 ^C [root@master01 zookeeper]# host 10.54.23.11 169.46.164.10.in-addr.arpa domain name pointer master01.sys76.com.

mike_bronson7 · ‎06-11-2018

what variable represented the - "enabled replication for this Postgres instance" ?

mike_bronson7 · ‎06-11-2018

@Geoffrey , files are created each day ( average 5 files each day ) , on other clusters this isnt happens , so if I will delete the old files then the problem will return , so I need to find the root cause , maybe conf problem but my cluster have the same conf of cluster that not have this problem , so I am little confuse ,

mike_bronson7 · ‎06-10-2018

I have found myself in a situation where the pg_xlog folder contains 80GB of data maybe because wrong configuration how to purge or cleanup the files under pg_xlog? /var/lib/pgsql/data/pg_xlog [root@master02 pg_xlog]# ls -ltr | more total 53887180 -rw-------. 1 postgres postgres 16777216 Sep 19 2017 0000000100000000000000F1 -rw-------. 1 postgres postgres 16777216 Sep 19 2017 0000000100000000000000F2 -rw------- 1 postgres postgres 16777216 Sep 19 2017 0000000100000000000000F3 -rw------- 1 postgres postgres 16777216 Sep 19 2017 0000000100000000000000F4 -rw------- 1 postgres postgres 16777216 Sep 19 2017 0000000100000000000000F5 -rw------- 1 postgres postgres 16777216 Sep 20 2017 0000000100000000000000F6 -rw------- 1 postgres postgres 16777216 Sep 20 2017 0000000100000000000000F7 -rw------- 1 postgres postgres 16777216 Sep 20 2017 0000000100000000000000F8 -rw------- 1 postgres postgres 16777216 Sep 20 2017 0000000100000000000000F9 -rw------- 1 postgres postgres 16777216 Sep 20 2017 0000000100000000000000FA -rw------- 1 postgres postgres 16777216 Sep 20 2017 0000000100000000000000FB -rw------- 1 postgres postgres 16777216 Sep 20 2017 0000000100000000000000FC -rw------- 1 postgres postgres 16777216 Sep 20 2017 0000000100000000000000FD -rw------- 1 postgres postgres 16777216 Sep 21 2017 0000000100000000000000FE -rw------- 1 postgres postgres 16777216 Sep 21 2017 000000010000000100000000 -rw------- 1 postgres postgres 16777216 Sep 21 2017 000000010000000100000001 -rw------- 1 postgres postgres 16777216 Sep 21 2017 000000010000000100000002 -rw------- 1 postgres postgres 16777216 Sep 21 2017 000000010000000100000003 -rw------- 1 postgres postgres 16777216 Sep 21 2017 000000010000000100000004 -rw------- 1 postgres postgres 16777216 Sep 21 2017 000000010000000100000005 -rw------- 1 postgres postgres 16777216 Sep 22 2017 000000010000000100000006 -rw------- 1 postgres postgres 16777216 Sep 22 2017 000000010000000100000007 -rw------- 1 postgres postgres 16777216 Sep 22 2017 000000010000000100000008 -rw------- 1 postgres postgres 16777216 Sep 22 2017 000000010000000100000009 -rw------- 1 postgres postgres 16777216 Sep 22 2017 00000001000000010000000A -rw------- 1 postgres postgres 16777216 Sep 22 2017 00000001000000010000000B -rw------- 1 postgres postgres 16777216 Sep 23 2017 00000001000000010000000C -rw------- 1 postgres postgres 16777216 Sep 23 2017 00000001000000010000000D -rw------- 1 postgres postgres 16777216 Sep 23 2017 00000001000000010000000E -rw------- 1 postgres postgres 16777216 Sep 23 2017 00000001000000010000000F -rw------- 1 postgres postgres 16777216 Sep 23 2017 000000010000000100000010 -rw------- 1 postgres postgres 16777216 Sep 23 2017 000000010000000100000011 -rw------- 1 postgres postgres 16777216 Sep 23 2017 000000010000000100000012 -rw------- 1 postgres postgres 16777216 Sep 24 2017 000000010000000100000013 -rw------- 1 postgres postgres 16777216 Sep 24 2017 000000010000000100000014 -rw------- 1 postgres postgres 16777216 Sep 24 2017 000000010000000100000015 -rw------- 1 postgres postgres 16777216 Sep 24 2017 000000010000000100000016 -rw------- 1 postgres postgres 16777216 Sep 24 2017 000000010000000100000017 -rw------- 1 postgres postgres 16777216 Sep 24 2017 000000010000000100000018 -rw------- 1 postgres postgres 16777216 Sep 24 2017 000000010000000100000019 -rw------- 1 postgres postgres 16777216 Sep 24 2017 00000001000000010000001A -rw------- 1 postgres postgres 16777216 Sep 24 2017 00000001000000010000001B -rw------- 1 postgres postgres 16777216 Sep 25 2017 00000001000000010000001C -rw------- 1 postgres postgres 16777216 Sep 25 2017 00000001000000010000001D -rw------- 1 postgres postgres 16777216 Sep 25 2017 00000001000000010000001E -rw------- 1 postgres postgres 16777216 Sep 25 2017 00000001000000010000001F -rw------- 1 postgres postgres 16777216 Sep 25 2017 000000010000000100000020 -rw------- 1 postgres postgres 16777216 Sep 25 2017 000000010000000100000021 --More--

mike_bronson7 · ‎06-10-2018

hi all we want to trigger the spark history deletion each hour ( spark.history.fs.cleaner.interval = 1h ) and to delete the spark history logs that older then one day ( spark.history.fs.cleaner.maxAge = 1d ) so we set the following: and restart the spark services but after 2 hours still logs not deleted in spite logs need to deleted after one hour according to our setings why logs not deleted after one hour ?

mike_bronson7 · ‎06-09-2018

I also found that, so what is the right variable to set ? ( JAVA_OPTS or JVMFLAGS ) export JVMFLAGS="-Djute.maxbuffer=11000000"

Online	Offline
Last Visited	‎08-27-2024 09:17 AM

Member Since	‎08-08-2017 09:40 AM
Last Visited	‎08-27-2024 09:17 AM
Posts	1,652
Kudos received	29

Cloudera Community

Re: how to find number of CPU core on datanode ma...

Re: postgresql + ambari server failed to open port...

Re: how to stop the thrift servers by REST API

Re: namenode is in safe mode

Re: Directory /grid/sdg/hadoop/hdfs/data became un...

Re: zookeper - Too many connections

Re: zookeper - Too many connections

Re: zookeper - Too many connections

Re: zookeper - Too many connections

zookeper - Too many connections

Re: How to clear /clean-up pg_xlog directory? on a...

Re: How to clear /clean-up pg_xlog directory? on a...

How to clear /clean-up pg_xlog directory? on ambar...

spark history logs not deleted

Re: how to increase the ZNode size in ambari clus...