Reply
Explorer
Posts: 10
Registered: ‎01-04-2015
Accepted Solution

Why Cloudera Manager create so many connections to Zookeeper?

hi,

 

In one of our CDH cluster (v5.0.0-beta1), zookeeper max connections are set to 60.

Now I found zookeeper logs out too many connections from Cloudera Manager.

 

view the connections by four-letter command 'cons',  output show as follows:

/192.168.56.80:36983[1](queued=0,recved=23475,sent=23475,sid=0x24ab7bb4e4d3abc,lop=PING,est=1420970377026,to=30000,lcxid=0x3d,lzxid=0x1d0002f281,lresp=1421204775170,llat=0,minlat=0,avglat=0,maxlat=17)
/192.168.56.80:35647[1](queued=0,recved=39836,sent=39836,sid=0x24ab7bb4e4d2724,lop=PING,est=1420806577434,to=30000,lcxid=0x3d,lzxid=0x1d0002f281,lresp=1421204770544,llat=0,minlat=0,avglat=0,maxlat=320)
/192.168.56.80:37019[1](queued=0,recved=14665,sent=14665,sid=0x24ab7bb4e4d457e,lop=PING,est=1421058577925,to=30000,lcxid=0x3d,lzxid=0x1d0002f281,lresp=1421204775172,llat=0,minlat=0,avglat=0,maxlat=39)

......

 

view Cloudera Manager log,  error information below occurs many times.

 

2015-01-12 11:50:32,445 ERROR [Parcel Update Service:components.ParcelDownloaderImpl@371] (3 skipped) Failed to download manifest. Status code: 404 URI: http://archive.cloudera.com/cdh5/parcels/latest/manifest.json

 

I wonder what conditions would cause Cloudera Manager to create massive connections to zookeeper.

and how do I solve it?

 

thanks in advance.

 

 

 

Cloudera Employee
Posts: 578
Registered: ‎01-20-2014

Re: Why Cloudera Manager create so many connections to Zookeeper?

There is a known issue where the Service Monitor in Cloudera Manager opens
connections to ZooKeeper but due to a bug, doesn't close the connection.
Your options are:
1. Disable HBase RegionServer Health Canary by changing the "HBase Region
Health Canary" enabled flag in the HBase service configuration to false.
This is the easiest solution.
2. Upgrade to a version of CM that includes the fix (4.8.5, 5.0.5, 5.1.4,
5.2.1, 5.3.0)
3. Frequently restart the service monitor (disruptive and not recommended)
BTW please try to move away from Beta versions. Our latest relase is v5.3.0
and it is highly recommended that you migrate to it

Regards,
Gautam Gopalakrishnan
Explorer
Posts: 10
Registered: ‎01-04-2015

Re: Why Cloudera Manager create so many connections to Zookeeper?

what I found just now proves your point, thank you very much.

New Contributor
Posts: 2
Registered: ‎07-22-2016

Re: Why Cloudera Manager create so many connections to Zookeeper?

Hi,

 

 I am having the same issues with CM 5.7.5.

And most of the connections coming from the cloudera manager nodes.

Explorer
Posts: 16
Registered: ‎09-12-2017

Re: Why Cloudera Manager create so many connections to Zookeeper?

I'm having the same problem on CM5.12.0, does this bug not fixed yet? 

Posts: 939
Kudos: 112
Solutions: 61
Registered: ‎04-06-2015

Re: Why Cloudera Manager create so many connections to Zookeeper?

@lwmonster If you read the previous comment from @GautamG the issue was resolved in a previous release. Unless something has changed this should be a seperate issue. I would suggest starting a new thread with further background on your specific situtaion. 



Cy Jervis, Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Explorer
Posts: 16
Registered: ‎09-12-2017

Re: Why Cloudera Manager create so many connections to Zookeeper?

hi, I have read the comment of @GautamG, I have tried the first suggestion, it do will not report Canary error on CM dashboard, but it still take all the connections of the machine which SERVICE MONITOR lies on.  My CM version is 5.12.0, almost the latest, but seems the bug still exists.

Explorer
Posts: 16
Registered: ‎09-12-2017

Re: Why Cloudera Manager create so many connections to Zookeeper?

Highlighted
Posts: 939
Kudos: 112
Solutions: 61
Registered: ‎04-06-2015

Re: Why Cloudera Manager create so many connections to Zookeeper?

I would suggest starting a new thread with all the details involved in your specific case. 



Cy Jervis, Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Announcements