Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Why Cloudera Manager create so many connections to Zookeeper?

Solved Go to solution

Why Cloudera Manager create so many connections to Zookeeper?

Explorer

hi,

 

In one of our CDH cluster (v5.0.0-beta1), zookeeper max connections are set to 60.

Now I found zookeeper logs out too many connections from Cloudera Manager.

 

view the connections by four-letter command 'cons',  output show as follows:

/192.168.56.80:36983[1](queued=0,recved=23475,sent=23475,sid=0x24ab7bb4e4d3abc,lop=PING,est=1420970377026,to=30000,lcxid=0x3d,lzxid=0x1d0002f281,lresp=1421204775170,llat=0,minlat=0,avglat=0,maxlat=17)
/192.168.56.80:35647[1](queued=0,recved=39836,sent=39836,sid=0x24ab7bb4e4d2724,lop=PING,est=1420806577434,to=30000,lcxid=0x3d,lzxid=0x1d0002f281,lresp=1421204770544,llat=0,minlat=0,avglat=0,maxlat=320)
/192.168.56.80:37019[1](queued=0,recved=14665,sent=14665,sid=0x24ab7bb4e4d457e,lop=PING,est=1421058577925,to=30000,lcxid=0x3d,lzxid=0x1d0002f281,lresp=1421204775172,llat=0,minlat=0,avglat=0,maxlat=39)

......

 

view Cloudera Manager log,  error information below occurs many times.

 

2015-01-12 11:50:32,445 ERROR [Parcel Update Service:components.ParcelDownloaderImpl@371] (3 skipped) Failed to download manifest. Status code: 404 URI: http://archive.cloudera.com/cdh5/parcels/latest/manifest.json

 

I wonder what conditions would cause Cloudera Manager to create massive connections to zookeeper.

and how do I solve it?

 

thanks in advance.

 

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Why Cloudera Manager create so many connections to Zookeeper?

There is a known issue where the Service Monitor in Cloudera Manager opens
connections to ZooKeeper but due to a bug, doesn't close the connection.
Your options are:
1. Disable HBase RegionServer Health Canary by changing the "HBase Region
Health Canary" enabled flag in the HBase service configuration to false.
This is the easiest solution.
2. Upgrade to a version of CM that includes the fix (4.8.5, 5.0.5, 5.1.4,
5.2.1, 5.3.0)
3. Frequently restart the service monitor (disruptive and not recommended)
BTW please try to move away from Beta versions. Our latest relase is v5.3.0
and it is highly recommended that you migrate to it

Regards,
Gautam Gopalakrishnan
8 REPLIES 8

Re: Why Cloudera Manager create so many connections to Zookeeper?

There is a known issue where the Service Monitor in Cloudera Manager opens
connections to ZooKeeper but due to a bug, doesn't close the connection.
Your options are:
1. Disable HBase RegionServer Health Canary by changing the "HBase Region
Health Canary" enabled flag in the HBase service configuration to false.
This is the easiest solution.
2. Upgrade to a version of CM that includes the fix (4.8.5, 5.0.5, 5.1.4,
5.2.1, 5.3.0)
3. Frequently restart the service monitor (disruptive and not recommended)
BTW please try to move away from Beta versions. Our latest relase is v5.3.0
and it is highly recommended that you migrate to it

Regards,
Gautam Gopalakrishnan

Re: Why Cloudera Manager create so many connections to Zookeeper?

Explorer

what I found just now proves your point, thank you very much.

Re: Why Cloudera Manager create so many connections to Zookeeper?

New Contributor

Hi,

 

 I am having the same issues with CM 5.7.5.

And most of the connections coming from the cloudera manager nodes.

Re: Why Cloudera Manager create so many connections to Zookeeper?

Explorer

I'm having the same problem on CM5.12.0, does this bug not fixed yet? 

Re: Why Cloudera Manager create so many connections to Zookeeper?

Community Manager

@lwmonster If you read the previous comment from @GautamG the issue was resolved in a previous release. Unless something has changed this should be a seperate issue. I would suggest starting a new thread with further background on your specific situtaion. 



Cy Jervis, Community Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:
Community Guidelines
How to use the forum

Re: Why Cloudera Manager create so many connections to Zookeeper?

Explorer

hi, I have read the comment of @GautamG, I have tried the first suggestion, it do will not report Canary error on CM dashboard, but it still take all the connections of the machine which SERVICE MONITOR lies on.  My CM version is 5.12.0, almost the latest, but seems the bug still exists.

Highlighted

Re: Why Cloudera Manager create so many connections to Zookeeper?

Explorer

Re: Why Cloudera Manager create so many connections to Zookeeper?

Community Manager

I would suggest starting a new thread with all the details involved in your specific case. 



Cy Jervis, Community Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:
Community Guidelines
How to use the forum
Don't have an account?
Coming from Hortonworks? Activate your account here