Member since
02-17-2025
6
Posts
0
Kudos Received
0
Solutions
03-07-2025
07:01 AM
Hi @Shelton @MattWho much much thanks to detailed information , i donot know how to express my gratitude , much thanks for helping i was so much struggling to get it . also i need one more information from you now i have 3 node cluster but i need to make it as production ready state by configuring SSDs and seperate repository for flow content and provenance so how to consider this or do configuration for production use case . presently it has default configuration and using 30gb and cluster is showing need help please how to configure SSD for production use case for existing cluster i found this blog https://community.cloudera.com/t5/Community-Articles/NiFi-Sizing-Guide-Deployment-Best-Practices/ta-p/246781 which gave me much insights but how to configure or some sort of examples i could not find . i wanted to have separate SSDs for all three repos and production ready so seeking your valuable insights here and if any blogs please suggest
... View more
Labels:
- Labels:
-
Apache NiFi
03-07-2025
06:36 AM
Hi @Shelton Thanks so much for detailed information @MattWho thanks much for the reply and apologies for short info. based on above information was able to create SSL certificates and generate Keystore and trustore in jks format . initially i was not configured CA file into truststore so faced some issue 2. then i did not added nifi nodes entries as intial identity in autherizers.xml file so above issue occured . i followed cloudera blogs where you had informed https://community.cloudera.com/t5/Support-Questions/insufficient-permissions-untrusted-proxy/m-p/366443#M239582 based on these i was able to resolve and 3 node cluster with external zookeeper was able to up. i appreciate your kind help and your time here . much thanks to both 🙂
... View more
03-04-2025
12:22 AM
Hi @Shelton Thanks a a Lot , i did troubleshooting steps as mentioned above and did restart of the all zookeeper and did some configuration changes like in nifi.properties file 1. nifi.zookeeper.connect.string=zookeepernode1.x.x.x.net:2181,zookeepernode2.x.x.x.net:2181,zookeepernode3.x.x.x.net:2181 2. nifi.zookeeper.ssl.client.auth=none 3. nifi.state.management.embedded.zookeeper.start=false in zookeeper nodes[1-3]: 1. dataDir=/var/lib/zookeeper from this location myid file was unable to fetch then we did restart of the VM post that it read it. 2. added server.1=zookeepernode1.x.x.x.net:2888:3888 server.2=zookeepernode2.x.x.x.net:2888:3888 server.3=zookeepernode3.x.x.x.net:2888:3888 since this is clustered setup this is must . Network configuration: 1. in Network manager enabled all ports like 2181 , 2888 and 3888 in all servers. with above changes made zookeeper to work and nifi is able to connect to external zookeeper cluster.
... View more
03-03-2025
03:27 AM
Hi @MattWho Need your help and suggestion here as i went through many of the Nifi related article here , since my use case is different need your valuable suggestion . 1. i want to know how to use SSL\TLS to have https mode for nifi url 2. in our organization already we are generating signed certificate with SAN entries as nifi-dev.x.x.net [domain we try to access via browser] nifinode1.x.x.net nifinode2.x.x.net nifinode3.x.x.net zookeepernode1.x.x.net zookeepernode2.x.x.net zookeepernode3.x.x.net post request raised we get a file nifi.crt file with this what else certificates are required\Needed? 3. as trial basis tried already creating keystore and truststore but seems some issue 4. 5.
... View more
Labels:
- Labels:
-
Apache NiFi
-
Apache Zookeeper
02-20-2025
12:17 AM
i have issue in connecting external zookeeper and nifi cluster , i have 3 nodes like zookeepernode1 zookeepernode2 zookeepernode3 and nifinode1 and nifinode2 nifiuser@zookeepernode3:~$ /opt/zookeeper/bin/zkServer.sh status /usr/bin/java ZooKeeper JMX enabled by default Using config: /opt/zookeeper/bin/../conf/zoo.cfg myid could not be determined, will not able to locate clientPort in the server configs. Client port found: 2181. Client address: localhost. Client SSL: false. Mode: follower root@zookeepernode1:~# /opt/zookeeper/bin/zkServer.sh status ZooKeeper JMX enabled by default Using config: /opt/zookeeper/bin/../conf/zoo.cfg Client port not found in static config file. Looking in dynamic config file. grep: : No such file or directory Client port not found in the server configs Client port found: 2181. Client address: localhost. Client SSL: false. Mode: follower root@zookeepernode2:/opt/zookeeper/logs# /opt/zookeeper/bin/zkServer.sh status ZooKeeper JMX enabled by default Using config: /opt/zookeeper/bin/../conf/zoo.cfg Client port not found in static config file. Looking in dynamic config file. grep: : No such file or directory Client port not found in the server configs Client port found: 2181. Client address: localhost. Client SSL: false. Mode: leader checking nifi logs [since both nodes logs are same so pasting logs from nifinode2] root@nifinode2:~# tail -f /opt/nifi/logs/nifi-app.log 2025-02-17 16:05:41,069 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is disconnected: [id: 0x41871c79 , L:/53.13.138.69:55258 ! R:zookeepernode2/53.13.138.72:2181] 2025-02-17 16:05:41,069 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is told closing 2025-02-17 16:05:41,111 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty SSL handler added for channel: [id: 0x29 83cd7a] 2025-02-17 16:05:41,113 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is connected: [id: 0x2983cd7a, L :/53.13.138.69:57908 - R:zookeepernode1/53.13.138.71:2181] 2025-02-17 16:05:41,114 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is disconnected: [id: 0x2983cd7a , L:/53.13.138.69:57908 ! R:zookeepernode1/53.13.138.71:2181] 2025-02-17 16:05:41,114 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is told closing 2025-02-17 16:05:41,157 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty SSL handler added for channel: [id: 0x80 2bf7fc] 2025-02-17 16:05:41,158 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is connected: [id: 0x802bf7fc, L :/53.13.138.69:38350 - R:zookeepernode3/53.13.247.198:2181] 2025-02-17 16:05:41,159 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is disconnected: [id: 0x802bf7fc , L:/53.13.138.69:38350 ! R:zookeepernode3/53.13.247.198:2181] 2025-02-17 16:05:41,159 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is told closing here are my configurations in Nifi.properties # zookeeper properties, used for cluster management # nifi.zookeeper.connect.string=zookeepernode1:2181,zookeepernode2:2181,zookeepernode3:2181 nifi.zookeeper.connect.timeout=10 secs nifi.zookeeper.session.timeout=10 secs nifi.zookeeper.root.node=/nifi root@nifinode1:/opt/nifi/conf# cat ./zookeeper.properties initLimit=10 autopurge.purgeInterval=24 syncLimit=5 tickTime=2000 dataDir=./state/zookeeper autopurge.snapRetainCount=30 server.1=zookeepernode1:2888:3888;2181 server.2=zookeepernode2:2888:3888;2181 server.3=zookeepernode3:2888:3888;2181 also i have myid file created properly in all 3 zookeeper nodes as 1 2 and 3 and there root@zookeepernode1:~# ls /var/lib/zookeeper/ myid version-2 root@zookeepernode1:~# cat /var/lib/zookeeper/myid 1 same but id will be 2 and 3 respectively in other nodes. please do help me here . your response will be appreciated
... View more
Labels:
- Labels:
-
Apache NiFi
-
Apache Zookeeper
02-17-2025
08:38 AM
i too have same issue , i have 3 nodes like zookeepernode1 zookeepernode2 zookeepernode3 nifiuser@zookeepernode3:~$ /opt/zookeeper/bin/zkServer.sh status /usr/bin/java ZooKeeper JMX enabled by default Using config: /opt/zookeeper/bin/../conf/zoo.cfg myid could not be determined, will not able to locate clientPort in the server configs. Client port found: 2181. Client address: localhost. Client SSL: false. Mode: follower in all three i get this status but in nifi root@nifinode2:~# tail -f /opt/nifi/logs/nifi-app.log 2025-02-17 16:05:41,069 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is disconnected: [id: 0x41871c79 , L:/53.13.138.69:55258 ! R:zookeepernode2/53.13.138.72:2181] 2025-02-17 16:05:41,069 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is told closing 2025-02-17 16:05:41,111 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty SSL handler added for channel: [id: 0x29 83cd7a] 2025-02-17 16:05:41,113 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is connected: [id: 0x2983cd7a, L :/53.13.138.69:57908 - R:zookeepernode1/53.13.138.71:2181] 2025-02-17 16:05:41,114 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is disconnected: [id: 0x2983cd7a , L:/53.13.138.69:57908 ! R:zookeepernode1/53.13.138.71:2181] 2025-02-17 16:05:41,114 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is told closing 2025-02-17 16:05:41,157 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty SSL handler added for channel: [id: 0x80 2bf7fc] 2025-02-17 16:05:41,158 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is connected: [id: 0x802bf7fc, L :/53.13.138.69:38350 - R:zookeepernode3/53.13.247.198:2181] 2025-02-17 16:05:41,159 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is disconnected: [id: 0x802bf7fc , L:/53.13.138.69:38350 ! R:zookeepernode3/53.13.247.198:2181] 2025-02-17 16:05:41,159 INFO [epollEventLoopGroup-4-1] o.apache.zookeeper.ClientCnxnSocketNetty channel is told closing here are my configurations # zookeeper properties, used for cluster management # nifi.zookeeper.connect.string=zookeepernode1:2181,zookeepernode2:2181,zookeepernode3:2181 nifi.zookeeper.connect.timeout=10 secs nifi.zookeeper.session.timeout=10 secs nifi.zookeeper.root.node=/nifi root@nifinode1:/opt/nifi/conf# cat ./zookeeper.properties initLimit=10 autopurge.purgeInterval=24 syncLimit=5 tickTime=2000 dataDir=./state/zookeeper autopurge.snapRetainCount=30 server.1=zookeepernode1:2888:3888;2181 server.2=zookeepernode2:2888:3888;2181 server.3=zookeepernode3:2888:3888;2181 also i have myid file created properly
... View more