Reply
Explorer
Posts: 6
Registered: ‎09-11-2015

CM server keeps outgoind connection open

We found out that CM server establishes some connections to the remote host in the internet and keeps them for a lifetime.

Remote address is:

151.101.36.167:443
151.101.36.167:80

 

What such connections are used for? (Send Diagnostic is turned off)

Cloudera Employee
Posts: 21
Registered: ‎01-18-2017

Re: CM server keeps outgoind connection open

Hello DemonBY,

 

This IP is a translation for archive.cloudera.com, would you please post the output of this command:

 

$ sudo ps ex | grep 151.101.36.167

 

Thanks!
Laith
Explorer
Posts: 6
Registered: ‎09-11-2015

Re: CM server keeps outgoind connection open

Your command returns nothing.

Here are some details

sudo netstat -anpl | grep 151.101.36.167
tcp        0      0 cm_host:48035          151.101.36.167:443          ESTABLISHED 2485/java           
tcp        0      0 cm_host:4734           151.101.36.167:80           ESTABLISHED 2485/java 

sudo ps -ef | grep 2485
496       2485  2483  5 Jun06 ?        11:58:21 /usr/java/jdk1.7.0_80/bin/java -cp .:lib/*:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar -server -Dlog4j.configuration=file:/etc/cloudera-scm-server/log4j.properties -Dfile.encoding=UTF-8 -Dcmf.root.logger=INFO,LOGFILE -Dcmf.log.dir=/var/log/cloudera-scm-server -Dcmf.log.file=cloudera-scm-server.log -Dcmf.jetty.threshhold=WARN -Dcmf.schema.dir=/usr/share/cmf/schema -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dpython.home=/usr/share/cmf/python -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+HeapDumpOnOutOfMemoryError -Xmx2G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:OnOutOfMemoryError=kill -9 %p com.cloudera.server.cmf.Main

 

New Contributor
Posts: 2
Registered: ‎07-06-2018

Re: CM server keeps outgoind connection open

Hi,

 

I have the same "issue" on my cluster cloudera (v5.14). Is it possible to have an explanation about this connections ? What information is sent and why ?

I didn't find anything about this in the documentation.

 

It's every hour and there are 2 differents IP : 104.16.19.10 and 151.101.36.167

 

 

root@serveur-hadoop:~# lsof -Pni | grep 151.101.36.167
java      21591 cloudera-scm  244u  IPv4 510376752      0t0  TCP X.X.X.X:58048->151.101.36.167:80 (ESTABLISHED)
java      21591 cloudera-scm  301u  IPv4 510376750      0t0  TCP X.X.X.X:38753->151.101.36.167:443 (ESTABLISHED)

 

 

Thank you.

Martin

Posts: 696
Topics: 1
Kudos: 162
Solutions: 87
Registered: ‎04-22-2014

Re: CM server keeps outgoind connection open

@martinbo,

 

I believe those connections are for archive.cloudera.com which is where Cloudera Parcels are stored.

I see the same behavior, but with a different IP address (most likely due to my different geographic location).

 

When Cloudera Manager looks for new parcels, it will open connections to the URLs specified in the Administration --> Settings --> Parcels --> Remote Parcel Repository URLs

 

By default, you may have a mixture of "https://" and "http://" urls listed in the Parcels configuration.  This is why you see connections to both ports 80 and 443.

 

I observed that the connections would remain for a few minutes after the check, but eventually the connections would be terminated.  I am not sure why these are open for a few minutes, but I confirmed via tcpdump that no traffic is active over these ports.

 

Another thing to check in your Parcels configuration is "Parcel Update Frequency".  If you are seeing connections to the parcel repositories open all the time, make sure it is set to 1 day or something bigger.   

 

-Ben

 

 

 

 

New Contributor
Posts: 2
Registered: ‎07-06-2018

Re: CM server keeps outgoind connection open

Hi Ben,

Thank you, this explains the connections. Indeed the "parcel update frequency" settings is actually 1 hour, I will set it.

Martin
Highlighted
Posts: 696
Topics: 1
Kudos: 162
Solutions: 87
Registered: ‎04-22-2014

Re: CM server keeps outgoind connection open

@martinbo,

 

No problem.

 

I should note that connecting to the archive servers is normal and is part of the periodic check for new parcels.

Announcements