Skip to main content
pavanshettyg5
pavanshettyg5
Silver ⭐️⭐️
February 20, 2025
Solved

External zookeeper and nifi cluster connection issue

i have issue in connecting external zookeeper and nifi cluster , i have 3 nodes like zookeepernode1 zookeepernode2 zookeepernode3 and nifinode1 and nifinode2


nifiuser@zookeepernode3:~$ /opt/zookeeper/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
myid could not be determined, will not able to locate clientPort in the server configs.
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower

root@zookeepernode1:~# /opt/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Client port not found in static config file. Looking in dynamic config file.
grep: : No such file or directory
Client port not found in the server configs
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower

root@zookeepernode2:/opt/zookeeper/logs# /opt/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Client port not found in static config file. Looking in dynamic config file.
grep: : No such file or directory
Client port not found in the server configs
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: leader

checking nifi logs [since both nodes logs are same so pasting logs from nifinode2]

root@nifinode2:~# tail -f /opt/nifi/logs/nifi-app.log
2025-02-17 16:05:41,069 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is disconnected: [id: 0x41871c79
, L:/53.13.138.69:55258 ! R:zookeepernode2/53.13.138.72:2181]
2025-02-17 16:05:41,069 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is told closing
2025-02-17 16:05:41,111 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty SSL handler added for channel: [id: 0x29
83cd7a]
2025-02-17 16:05:41,113 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is connected: [id: 0x2983cd7a, L
:/53.13.138.69:57908 - R:zookeepernode1/53.13.138.71:2181]
2025-02-17 16:05:41,114 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is disconnected: [id: 0x2983cd7a
, L:/53.13.138.69:57908 ! R:zookeepernode1/53.13.138.71:2181]
2025-02-17 16:05:41,114 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is told closing
2025-02-17 16:05:41,157 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty SSL handler added for channel: [id: 0x80
2bf7fc]
2025-02-17 16:05:41,158 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is connected: [id: 0x802bf7fc, L
:/53.13.138.69:38350 - R:zookeepernode3/53.13.247.198:2181]
2025-02-17 16:05:41,159 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is disconnected: [id: 0x802bf7fc
, L:/53.13.138.69:38350 ! R:zookeepernode3/53.13.247.198:2181]
2025-02-17 16:05:41,159 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is told closing

here are my configurations in Nifi.properties
# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=zookeepernode1:2181,zookeepernode2:2181,zookeepernode3:2181
nifi.zookeeper.connect.timeout=10 secs
nifi.zookeeper.session.timeout=10 secs
nifi.zookeeper.root.node=/nifi

root@nifinode1:/opt/nifi/conf# cat ./zookeeper.properties
initLimit=10
autopurge.purgeInterval=24
syncLimit=5
tickTime=2000
dataDir=./state/zookeeper
autopurge.snapRetainCount=30
server.1=zookeepernode1:2888:3888;2181
server.2=zookeepernode2:2888:3888;2181
server.3=zookeepernode3:2888:3888;2181

also i have myid file created properly in all 3 zookeeper nodes as 1 2 and 3 and there 

root@zookeepernode1:~# ls /var/lib/zookeeper/
myid version-2
root@zookeepernode1:~# cat /var/lib/zookeeper/myid
1
same but id will be 2 and 3 respectively in other nodes.
please do help me here . your response will be appreciated 

 

Shelton
SheltonAnswer
Gold ⭐️⭐️⭐️
March 3, 2025

@pavanshettyg5 
But looking at the error messages when running zkServer.sh status on the ZooKeeper nodes: on zookeepernode1 and 2, there's a message: "Client port not found in static config file. Looking in dynamic config file." Then a grep error: "grep: : No such file or directory". This suggests that the static zoo.cfg is missing the clientPort entry, and the dynamic configuration file (which is probably specified via dynamicConfigFile in zoo.cfg) is either not present or misconfigured.

To resolve the ZooKeeper and NiFi connectivity issues, follow these steps and hopefully that will resolve your nifi connectivity issue

Step 1: Configure ZooKeeper to Bind to All Interfaces

Problem: ZooKeeper nodes are binding to localhost, preventing remote connections from NiFi.
Fix: Update zoo.cfg on each ZooKeeper node to bind to 0..0.0.0 (all interfaces).

1. Edit zoo.cfg on each ZooKeeper node:

Spoiler
vi /opt/zookeeper/conf/zoo.cfg

2. Add/Modify these lines

Spoiler
clientPort=2181
clientPortAddress=0.0.0.0

3. Restart ZooKeeper on each node:

Spoiler
/opt/zookeeper/bin/zkServer.sh restart

Step 2: Verify ZooKeeper Configuration

After restarting, check the status

Spoiler
/opt/zookeeper/bin/zkServer.sh status

Expected Output:
Client address: 0.0.0.0(not localhost).
One node should beleader, others follower.

Step 3: Check ZooKeeper Network Connectivity

From NiFi nodes, test connectivity to ZooKeeper:

Spoiler
telnet zookeepernode1 2181
telnet zookeepernode2 2181
telnet zookeepernode3 2181

  • If connections fail, check firewalls/security groups to allow traffic on port 2181.

Step 4: Validate ZooKeeper Dynamic Configuration (If Applicable)

If using dynamic reconfiguration:

1. Ensure the dynamic config file (e.g., zoo_dynamic.cfg) has entries like

Spoiler
server.1=zookeepernode1:2888:3888:participant;zookeepernode1:2181
server.2=zookeepernode2:2888:3888:participant;zookeepernode2:2181
server.3=zookeepernode3:2888:3888:participant;zookeepernode3:2181

2. Confirm the static zoo.cfg references the dynamic file:

Spoiler
dynamicConfigFile=/opt/zookeeper/conf/zoo_dynamic.cfg

Step 5: Verify NiFi Configuration

Ensure nifi.properties points to the correct ZooKeeper ensemble:

Spoiler
nifi.zookeeper.connect.string=zookeepernode1:2181,zookeepernode2:2181,zookeepernode3:2181

Step 6: Restart NiFi Services

Restart NiFi on all nodes:

Spoiler
/opt/nifi/bin/nifi.sh restart

Check logs for successful connections:

Spoiler
tail -f /opt/nifi/logs/nifi-app.log

Troubleshooting Summary

  • ZooKeeper Binding: Ensure ZooKeeper listens on 0.0.0.0:2181, not localhost.

  • Firewall Rules: Allow traffic between NiFi and ZooKeeper nodes on ports 2181, 2888, and 3888.

  • Hostname Resolution: Confirm zookeepernode1, zookeepernode2, and zookeepernode3 resolve to correct IPs on NiFi nodes.

By addressing ZooKeeper's binding configuration and network accessibility, NiFi should successfully connect to the ZooKeeper cluster.

Happy hadooping

View in thread

2 replies

Shelton
SheltonAnswer
Gold ⭐️⭐️⭐️
March 3, 2025

@pavanshettyg5 
But looking at the error messages when running zkServer.sh status on the ZooKeeper nodes: on zookeepernode1 and 2, there's a message: "Client port not found in static config file. Looking in dynamic config file." Then a grep error: "grep: : No such file or directory". This suggests that the static zoo.cfg is missing the clientPort entry, and the dynamic configuration file (which is probably specified via dynamicConfigFile in zoo.cfg) is either not present or misconfigured.

To resolve the ZooKeeper and NiFi connectivity issues, follow these steps and hopefully that will resolve your nifi connectivity issue

Step 1: Configure ZooKeeper to Bind to All Interfaces

Problem: ZooKeeper nodes are binding to localhost, preventing remote connections from NiFi.
Fix: Update zoo.cfg on each ZooKeeper node to bind to 0..0.0.0 (all interfaces).

1. Edit zoo.cfg on each ZooKeeper node:

Spoiler
vi /opt/zookeeper/conf/zoo.cfg

2. Add/Modify these lines

Spoiler
clientPort=2181
clientPortAddress=0.0.0.0

3. Restart ZooKeeper on each node:

Spoiler
/opt/zookeeper/bin/zkServer.sh restart

Step 2: Verify ZooKeeper Configuration

After restarting, check the status

Spoiler
/opt/zookeeper/bin/zkServer.sh status

Expected Output:
Client address: 0.0.0.0(not localhost).
One node should beleader, others follower.

Step 3: Check ZooKeeper Network Connectivity

From NiFi nodes, test connectivity to ZooKeeper:

Spoiler
telnet zookeepernode1 2181
telnet zookeepernode2 2181
telnet zookeepernode3 2181

  • If connections fail, check firewalls/security groups to allow traffic on port 2181.

Step 4: Validate ZooKeeper Dynamic Configuration (If Applicable)

If using dynamic reconfiguration:

1. Ensure the dynamic config file (e.g., zoo_dynamic.cfg) has entries like

Spoiler
server.1=zookeepernode1:2888:3888:participant;zookeepernode1:2181
server.2=zookeepernode2:2888:3888:participant;zookeepernode2:2181
server.3=zookeepernode3:2888:3888:participant;zookeepernode3:2181

2. Confirm the static zoo.cfg references the dynamic file:

Spoiler
dynamicConfigFile=/opt/zookeeper/conf/zoo_dynamic.cfg

Step 5: Verify NiFi Configuration

Ensure nifi.properties points to the correct ZooKeeper ensemble:

Spoiler
nifi.zookeeper.connect.string=zookeepernode1:2181,zookeepernode2:2181,zookeepernode3:2181

Step 6: Restart NiFi Services

Restart NiFi on all nodes:

Spoiler
/opt/nifi/bin/nifi.sh restart

Check logs for successful connections:

Spoiler
tail -f /opt/nifi/logs/nifi-app.log

Troubleshooting Summary

  • ZooKeeper Binding: Ensure ZooKeeper listens on 0.0.0.0:2181, not localhost.

  • Firewall Rules: Allow traffic between NiFi and ZooKeeper nodes on ports 2181, 2888, and 3888.

  • Hostname Resolution: Confirm zookeepernode1, zookeepernode2, and zookeepernode3 resolve to correct IPs on NiFi nodes.

By addressing ZooKeeper's binding configuration and network accessibility, NiFi should successfully connect to the ZooKeeper cluster.

Happy hadooping

Translate
pavanshettyg5
pavanshettyg5Author
Silver ⭐️⭐️
March 4, 2025

Hi @Shelton 
Thanks a a Lot , i did troubleshooting steps as mentioned above and did restart of the all zookeeper and did some configuration changes like in nifi.properties file
1. nifi.zookeeper.connect.string=zookeepernode1.x.x.x.net:2181,zookeepernode2.x.x.x.net:2181,zookeepernode3.x.x.x.net:2181
2.

nifi.zookeeper.ssl.client.auth=none
3.
nifi.state.management.embedded.zookeeper.start=false

in zookeeper nodes[1-3]:
1. dataDir=/var/lib/zookeeper from this location myid file was unable to fetch then we did restart of the VM post that it read it.
2. added 
server.1=zookeepernode1.x.x.x.net:2888:3888
server.2=zookeepernode2.x.x.x.net:2888:3888
server.3=zookeepernode3.x.x.x.net:2888:3888
since this is clustered setup this is must .

Network configuration:
1. in Network manager enabled all ports like 2181 , 2888 and 3888 in all servers.
with above changes made zookeeper to work and nifi is able to connect to external zookeeper cluster.
Translate
Terms & ConditionsAccessibility statement