Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Why Cloudera Manager create so many connections to Zookeeper?

avatar
Contributor

hi,

 

In one of our CDH cluster (v5.0.0-beta1), zookeeper max connections are set to 60.

Now I found zookeeper logs out too many connections from Cloudera Manager.

 

view the connections by four-letter command 'cons',  output show as follows:

/192.168.56.80:36983[1](queued=0,recved=23475,sent=23475,sid=0x24ab7bb4e4d3abc,lop=PING,est=1420970377026,to=30000,lcxid=0x3d,lzxid=0x1d0002f281,lresp=1421204775170,llat=0,minlat=0,avglat=0,maxlat=17)
/192.168.56.80:35647[1](queued=0,recved=39836,sent=39836,sid=0x24ab7bb4e4d2724,lop=PING,est=1420806577434,to=30000,lcxid=0x3d,lzxid=0x1d0002f281,lresp=1421204770544,llat=0,minlat=0,avglat=0,maxlat=320)
/192.168.56.80:37019[1](queued=0,recved=14665,sent=14665,sid=0x24ab7bb4e4d457e,lop=PING,est=1421058577925,to=30000,lcxid=0x3d,lzxid=0x1d0002f281,lresp=1421204775172,llat=0,minlat=0,avglat=0,maxlat=39)

......

 

view Cloudera Manager log,  error information below occurs many times.

 

2015-01-12 11:50:32,445 ERROR [Parcel Update Service:components.ParcelDownloaderImpl@371] (3 skipped) Failed to download manifest. Status code: 404 URI: http://archive.cloudera.com/cdh5/parcels/latest/manifest.json

 

I wonder what conditions would cause Cloudera Manager to create massive connections to zookeeper.

and how do I solve it?

 

thanks in advance.

 

 

 

1 ACCEPTED SOLUTION

avatar
There is a known issue where the Service Monitor in Cloudera Manager opens
connections to ZooKeeper but due to a bug, doesn't close the connection.
Your options are:
1. Disable HBase RegionServer Health Canary by changing the "HBase Region
Health Canary" enabled flag in the HBase service configuration to false.
This is the easiest solution.
2. Upgrade to a version of CM that includes the fix (4.8.5, 5.0.5, 5.1.4,
5.2.1, 5.3.0)
3. Frequently restart the service monitor (disruptive and not recommended)
BTW please try to move away from Beta versions. Our latest relase is v5.3.0
and it is highly recommended that you migrate to it

Regards,
Gautam Gopalakrishnan

View solution in original post

8 REPLIES 8

avatar
There is a known issue where the Service Monitor in Cloudera Manager opens
connections to ZooKeeper but due to a bug, doesn't close the connection.
Your options are:
1. Disable HBase RegionServer Health Canary by changing the "HBase Region
Health Canary" enabled flag in the HBase service configuration to false.
This is the easiest solution.
2. Upgrade to a version of CM that includes the fix (4.8.5, 5.0.5, 5.1.4,
5.2.1, 5.3.0)
3. Frequently restart the service monitor (disruptive and not recommended)
BTW please try to move away from Beta versions. Our latest relase is v5.3.0
and it is highly recommended that you migrate to it

Regards,
Gautam Gopalakrishnan

avatar
Contributor

what I found just now proves your point, thank you very much.

avatar
New Contributor

Hi,

 

 I am having the same issues with CM 5.7.5.

And most of the connections coming from the cloudera manager nodes.

avatar
Explorer

I'm having the same problem on CM5.12.0, does this bug not fixed yet? 

avatar
Community Manager

@lwmonster If you read the previous comment from @GautamG the issue was resolved in a previous release. Unless something has changed this should be a seperate issue. I would suggest starting a new thread with further background on your specific situtaion. 


Cy Jervis, Manager, Community Program
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar
Explorer

hi, I have read the comment of @GautamG, I have tried the first suggestion, it do will not report Canary error on CM dashboard, but it still take all the connections of the machine which SERVICE MONITOR lies on.  My CM version is 5.12.0, almost the latest, but seems the bug still exists.

avatar
Explorer

avatar
Community Manager

I would suggest starting a new thread with all the details involved in your specific case. 


Cy Jervis, Manager, Community Program
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.