question Re: Why Cloudera Manager create so many connections to Zookeeper? in Support Questions

Why Cloudera Manager create so many connections to Zookeeper?

Zhuo.Chen — Fri, 16 Sep 2022 09:19:01 GMT

hi,

In one of our CDH cluster (v5.0.0-beta1), zookeeper max connections are set to 60.

Now I found zookeeper logs out too many connections from Cloudera Manager.

view the connections by four-letter command 'cons', output show as follows:

/192.168.56.80:36983[1](queued=0,recved=23475,sent=23475,sid=0x24ab7bb4e4d3abc,lop=PING,est=1420970377026,to=30000,lcxid=0x3d,lzxid=0x1d0002f281,lresp=1421204775170,llat=0,minlat=0,avglat=0,maxlat=17)
/192.168.56.80:35647[1](queued=0,recved=39836,sent=39836,sid=0x24ab7bb4e4d2724,lop=PING,est=1420806577434,to=30000,lcxid=0x3d,lzxid=0x1d0002f281,lresp=1421204770544,llat=0,minlat=0,avglat=0,maxlat=320)
/192.168.56.80:37019[1](queued=0,recved=14665,sent=14665,sid=0x24ab7bb4e4d457e,lop=PING,est=1421058577925,to=30000,lcxid=0x3d,lzxid=0x1d0002f281,lresp=1421204775172,llat=0,minlat=0,avglat=0,maxlat=39)

......

view Cloudera Manager log, error information below occurs many times.

2015-01-12 11:50:32,445 ERROR [Parcel Update Service:components.ParcelDownloaderImpl@371] (3 skipped) Failed to download manifest. Status code: 404 URI: http://archive.cloudera.com/cdh5/parcels/latest/manifest.json

I wonder what conditions would cause Cloudera Manager to create massive connections to zookeeper.

and how do I solve it?

thanks in advance.

Re: Why Cloudera Manager create so many connections to Zookeeper?

GautamG — Wed, 14 Jan 2015 03:46:50 GMT

There is a known issue where the Service Monitor in Cloudera Manager opens
connections to ZooKeeper but due to a bug, doesn't close the connection.
Your options are:
1. Disable HBase RegionServer Health Canary by changing the "HBase Region
Health Canary" enabled flag in the HBase service configuration to false.
This is the easiest solution.
2. Upgrade to a version of CM that includes the fix (4.8.5, 5.0.5, 5.1.4,
5.2.1, 5.3.0)
3. Frequently restart the service monitor (disruptive and not recommended)
BTW please try to move away from Beta versions. Our latest relase is v5.3.0
and it is highly recommended that you migrate to it

Re: Why Cloudera Manager create so many connections to Zookeeper?

Zhuo.Chen — Wed, 14 Jan 2015 04:32:56 GMT

what I found just now proves your point, thank you very much.

Re: Why Cloudera Manager create so many connections to Zookeeper?

satish21 — Wed, 14 Dec 2016 08:08:01 GMT

Hi,

I am having the same issues with CM 5.7.5.

And most of the connections coming from the cloudera manager nodes.

Re: Why Cloudera Manager create so many connections to Zookeeper?

lwmonster — Fri, 15 Sep 2017 04:02:46 GMT

I'm having the same problem on CM5.12.0, does this bug not fixed yet?

Re: Why Cloudera Manager create so many connections to Zookeeper?

cjervis — Fri, 15 Sep 2017 12:53:42 GMT

@lwmonster If you read the previous comment from @GautamG the issue was resolved in a previous release. Unless something has changed this should be a seperate issue. I would suggest starting a new thread with further background on your specific situtaion.

Re: Why Cloudera Manager create so many connections to Zookeeper?

lwmonster — Mon, 18 Sep 2017 01:41:30 GMT

hi, I have read the comment of @GautamG, I have tried the first suggestion, it do will not report Canary error on CM dashboard, but it still take all the connections of the machine which SERVICE MONITOR lies on. My CM version is 5.12.0, almost the latest, but seems the bug still exists.

Re: Why Cloudera Manager create so many connections to Zookeeper?

lwmonster — Wed, 20 Sep 2017 08:47:08 GMT

@cjervis

Re: Why Cloudera Manager create so many connections to Zookeeper?

cjervis — Wed, 20 Sep 2017 12:16:06 GMT

I would suggest starting a new thread with all the details involved in your specific case.