Why Cloudera Manager create so many connections to Zookeeper?

[#1]

hi,

In one of our CDH cluster (v5.0.0-beta1), zookeeper max connections are set to 60.

Now I found zookeeper logs out too many connections from Cloudera Manager.

view the connections by four-letter command 'cons',  output show as follows:

/192.168.56.80:36983[1](queued=0,recved=23475,sent=23475,sid=0x24ab7bb4e4d3abc,lop=PING,est=1420970377026,to=30000,lcxid=0x3d,lzxid=0x1d0002f281,lresp=1421204775170,llat=0,minlat=0,avglat=0,maxlat=17)
/192.168.56.80:35647[1](queued=0,recved=39836,sent=39836,sid=0x24ab7bb4e4d2724,lop=PING,est=1420806577434,to=30000,lcxid=0x3d,lzxid=0x1d0002f281,lresp=1421204770544,llat=0,minlat=0,avglat=0,maxlat=320)
/192.168.56.80:37019[1](queued=0,recved=14665,sent=14665,sid=0x24ab7bb4e4d457e,lop=PING,est=1421058577925,to=30000,lcxid=0x3d,lzxid=0x1d0002f281,lresp=1421204775172,llat=0,minlat=0,avglat=0,maxlat=39)

......

view Cloudera Manager log,  error information below occurs many times.

2015-01-12 11:50:32,445 ERROR [Parcel Update Service:components.ParcelDownloaderImpl@371] (3 skipped) Failed to download manifest. Status code: 404 URI: http://archive.cloudera.com/cdh5/parcels/latest/manifest.json

I wonder what conditions would cause Cloudera Manager to create massive connections to zookeeper.

and how do I solve it?

thanks in advance.

[#2 · Accepted solution]

There is a known issue where the Service Monitor in Cloudera Manager opens
connections to ZooKeeper but due to a bug, doesn't close the connection.
Your options are:
1. Disable HBase RegionServer Health Canary by changing the "HBase Region
Health Canary" enabled flag in the HBase service configuration to false.
This is the easiest solution.
2. Upgrade to a version of CM that includes the fix (4.8.5, 5.0.5, 5.1.4,
5.2.1, 5.3.0)
3. Frequently restart the service monitor (disruptive and not recommended)
BTW please try to move away from Beta versions. Our latest relase is v5.3.0
and it is highly recommended that you migrate to it

[#3]

what I found just now proves your point, thank you very much.

[#4]

Hi,

 I am having the same issues with CM 5.7.5.

And most of the connections coming from the cloudera manager nodes.

[#5]

I'm having the same problem on CM5.12.0, does this bug not fixed yet? 

[#6]

@lwmonster If you read the previous comment from @GautamG the issue was resolved in a previous release. Unless something has changed this should be a seperate issue. I would suggest starting a new thread with further background on your specific situtaion. 

[#7]

hi, I have read the comment of @GautamG, I have tried the first suggestion, it do will not report Canary error on CM dashboard, but it still take all the connections of the machine which SERVICE MONITOR lies on.  My CM version is 5.12.0, almost the latest, but seems the bug still exists.

[#8]

@cjervis 

[#9]

I would suggest starting a new thread with all the details involved in your specific case.