Zk clientPort is using by other application why ?

[#1]

we have ambari cluster with 3 masters machines , on master01 and master03 Zk is running

as all know on master02 Zk clientPort use the value 2181 ,

and we seen from the logs that Zk not start because port is already in use

as the following ( on master02 )

netstat -tnlpa | grep 2181
tcp        0      0 10.164.28.152:38219     10.164.28.153:2181      ESTABLISHED 51471/java
tcp        0      0 10.164.28.152:38218     10.164.28.153:2181      ESTABLISHED 51471/java
tcp        0      0 10.164.28.152:38707     10.164.27.162:2181      ESTABLISHED 51471/java
tcp6       0      0 :::2181                 :::*                    LISTEN      24847/java
tcp6       0      0 10.164.28.152:2181      10.164.28.152:48270     ESTABLISHED 24847/java
tcp6       0      0 10.164.28.152:2181      10.164.28.152:40876     ESTABLISHED 24847/java
tcp6       0      0 10.164.28.152:39342     10.164.27.162:2181      ESTABLISHED 39008/java
tcp6       0      0 10.164.28.152:2181      10.164.27.162:53094     ESTABLISHED 24847/java
tcp6       0      0 10.164.28.152:37998     10.164.28.153:2181      ESTABLISHED 38168/java
tcp6       0      0 10.164.28.152:45319     10.164.28.153:2181      ESTABLISHED 39008/java
tcp6       0      0 10.164.28.152:39133     10.164.28.153:2181      ESTABLISHED 39008/java
tcp6       0      0 10.164.28.152:2181      10.164.28.153:33491     ESTABLISHED 24847/java
tcp6       0      0 10.164.28.152:37092     10.164.28.153:2181      ESTABLISHED 48592/java
tcp6       0      0 10.164.28.152:41413     10.164.28.153:2181      ESTABLISHED 28226/java
tcp6       0      0 10.164.28.152:40876     10.164.28.152:2181      ESTABLISHED 38168/java

so my question is

is it possible to work around on this problem to use other port as - 2182 , just to start the Zk on master02 machine ?

or maybe other suggestion how to start the Zk server on master02 machine ?

second , how it can be that Java in ambari cluster use this port that is allocated to Zk , why this happened ?

[#2 · Accepted solution]

@Michael Bronson

Regarding your query

1. is it possible to work around on this problem to use other port as - 2182 , just to start the Zk on master02 machine ?

>>> Yes, there is a concept of "config groups" in ambari which allows us to specify slightly different configurations for the same components installed on different host of the cluster. https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-operations/content/using_host_con...

So using config groups it is possible to define port 2182 for master02 host.

.

2. how it can be that Java in ambari cluster use this port that is allocated to Zk , why this happened ?

>>> Ideally each HDP components are defined in such a way that they do not cause any port conflict until there is an issue (mis configuration) or some other process is using the HDP component specific port already.

So please check which process is using the port 2181 on master02 node and then kill it and then try starting the Zookeeper. Like in this case we need to check which process is 24847 and why it is configured to use Zookeepers default port 2181

# ps -ef | grep 24847
# kill -9 24847        (Only if you are sure that this process is incorrectly configured and should nto be using port 2181)

.

[#3]

@Jay one note - but if I will kill the process it will affected on other services , so killing all them will showdown 3-4 services from ambari cluster , so my quastion is how to avoid the appliaction not to use the port that alocated already

[#4]

@Michael Bronson

Correct. So first it is better to get more information about that process which is using 2181 port. And then we will see it's configuration to know why it is configured to use that port.

# ps -ef | grep 24847

.

[#5]

one of them is - ambari-metrics-collector service for example

[#6]

@Jay what about just set the port 2182 in the "Port for running ZK Server" instead 2181 , this isn't

elegant solution but just to start the Zk on node - master02 , it will works?

[#7]

@Michael Bronson

Yes, we can change the default ZK port 2181 to something else for all the hosts. That will also be a easily managable option.