nifi with external zookeeper errors

[#1]

I have installed nifi on 3 hosts and zookeeper on one, for now.  In the nifi-app.log, I keep seeing these errors.

org.apache.nifi.priority.Rul;esManager Exception occurred when trying to update rules from teh stateManageProvider.  java.io.  Failed to obtain value from zookeeper for component with ID RulesManagerID with the exception code CONNECTIONLOSS.  I'm running zookeeper-3.4.13 until I can get a later release.  

In addition, in my zookeeper.out log, These messages keep rolling in the log:

Accepted socket connection from /IP/port

Exception causing close of sessions 0x0 Len error 369296128

Closed socket connection for client /IP/port (no session established for client)

Any help would be greatly appreciated.

Cathy

[#2]

@luv4diamonds Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @SAMSAL @cotopaul  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3]

Question was not answered.

[#4]

@MattWho Any insights here by any chance? Thanks!

[#5]

@luv4diamonds,

I do not have much experience with zookeeper but if you are running a 3 node nifi cluster and you have installed zookeeper on one of those nodes, why don't you use the embedded zookeeper directly? It would make sense to have your external zookeeper made totally external as right now you are assigning resources to your zookeeper instance as well. For example if you will have a problem with the node on which you have your zookeeper client, the entire cluster will be affected, whereas if you had your zookeeper on a different machine, the other nodes would not be affected. This is just an opinion 🙂

Also, please share some light on what NiFi Version you are using and how you configured your properties? I am talking here about nifi.properties, zookeeper.properties and state-management.xml.

[#6]

@luv4diamonds 

NiFi will NOT form a cluster with a single instance Zookeeper.  Zookeeper must have quorum otherwise a cluster coordinator and primary node will not get elected for NiFi.  For Quorum you should have an odd number of ZK nodes with a minimum of 3.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

[#7]

I'm running nifi-11.14.0.  I switched to embedded on my cluster and I'm still getting connection refused to port 2181 for only one of the nifi's.  In addition, I'm seeing this error.  KeeperErrorCode - ConnectionLoss for /nifi/components/RulesManagerID.  Any other ideas?

This messages did go away:  Exception causing close of sessions 0x0 Len error 369296128. 

Also, I can't share my files on here.

My state-management.xml

Connect String:  >zoo.mydomain:2181,zoo2.mydomain:2181,zoo3.mydomain:2181<

zookeepers.properties:

I have ssl configured and
server.1-zoo.mydomain:2888:3888;2181

server.1-zoo2.mydomain:2888:3888;2181

server.1-zoo3.mydomain:2888:3888;2181

nifi.properties embedded zookeeper: true

nifi.zookeeper.connect.string=zoo.mydomain:2181,zoo2.mydomain:2181,zoo3.mydomain:2181

[#8]

@luv4diamonds 
On my instances where I am running the embedded zookeeper, I am using port 2182 and not 2181. Maybe you can try and change it like that and test.
In addition, I assume that you have the file myid generated on every node, right?(https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.1/nifi-state-management/content/embedded_zookeep...)

[#9]

Thank you for replying.  I have tried so many different ports and I am even trying external zookeeper.  None of my nifi's can connect to any port I provide them.  It's almost like something is wrong with the code.  I have installed ZK on one of my actual nifi servers and that starts up immediately on port 2181.  That's what is leading me to think it's something in the code.  The other odd thing I keep seeing is, apache.zookeeper.ClientCnxnSocketNetty future isn't success.