I'm using NiFi 1.6.0, in a 3 node cluster. When I use GETSFTP (Set to ALL Nodes) in a clustered nifi the cluster seems to distribute the data acquired evenly among nodes. Does this mean that all 3 servers GetSFTP the data evenly? I also tried using FETCH SFTP to get the listings and then did a site to site, back to my own cluster and It did NOT distribute the Fetch 0 byte files evenly among the nodes for the fetch SFTP load to be evenly distributed. What would be the best practice to Load Balance SFTPGET in a nifi cluster? John ... View more

So I have 4 740xd Servers with 80 Threads on each and 384GB of ram each. How should I best handle heap for each nifi VM on the server? Option 1: 1, VM 80 cpus, NiFi set to 300GB of Ram for heap? And have the 4 nifis clustered. Option 2: 10, VMs 8 cpus each, NiFi set to 32GB of Ram for heap for each VM, These NiFis would also be clustered, across the Virtualized 740 and across the 4 hardware 740s Note: all these files going through these nifis are less than 4GB, and 90% of them are under 10MB ALSO what generation should I use for Java Garbage Collection? Attached are two pictures explaining my options. John ... View more

Hi Everyone! I keep getting a: ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss My EDITABLE config files are here in the associated directories in my google drive below and attached, this is a 3 node cluster: https://drive.google.com/drive/folders/11xM-sz8mUvpaiOOS4aiZ94TQGHHzConF?usp=sharing I made sure FirewallD was off, all ports used are free, and I followed these guides: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.1/bk_administration/content/clustering.html AND https://community.hortonworks.com/articles/135820/configuring-an-external-zookeeper-to-work-with-apa.html Any help would be greatly appreciated!! Here's my configs in plain text for reference: ###################################################10.0.0.89 Server###################################################### ####ZooKeeper.properties File##### clientPort=24489 initLimit=10 autopurge.purgeInterval=24 syncLimit=5 tickTime=2000 dataDir=./state/zookeeper autopurge.snapRetainCount=30 server.1=10.0.0.89:2888:3888 server.2=10.0.0.227:2888:3888 server.3=10.0.0.228:2888:3888 ####nifi.properties cluster section#### # cluster common properties (all nodes must have same values) # nifi.cluster.protocol.heartbeat.interval=5 sec nifi.cluster.protocol.is.secure=false # cluster node properties (only configure for cluster nodes) # nifi.cluster.is.node=true nifi.cluster.node.address=10.0.0.89 nifi.cluster.node.protocol.port=24489 nifi.cluster.node.protocol.threads=10 nifi.cluster.node.protocol.max.threads=50 nifi.cluster.node.event.history.size=25 nifi.cluster.node.connection.timeout=5 sec nifi.cluster.node.read.timeout=5 sec nifi.cluster.node.max.concurrent.requests=100 nifi.cluster.firewall.file= nifi.cluster.flow.election.max.wait.time=5 mins nifi.cluster.flow.election.max.candidates=3 # zookeeper properties, used for cluster management # nifi.zookeeper.connect.string=10.0.0.89:24489,10.0.0.227:24427,10.0.0.28:24428 nifi.zookeeper.connect.timeout=3 secs nifi.zookeeper.session.timeout=3 secs nifi.zookeeper.root.node=/nifi ###################################################################10.0.0.227 Server################################################################################################ ####ZooKeeper.properties File##### clientPort=24427 initLimit=10 autopurge.purgeInterval=24 syncLimit=5 tickTime=2000 dataDir=./state/zookeeper autopurge.snapRetainCount=30 server.1=10.0.0.89:2888:3888 server.2=10.0.0.227:2888:3888 server.3=10.0.0.228:2888:3888 #####nifi.properties cluster section ################### # cluster common properties (all nodes must have same values) # nifi.cluster.protocol.heartbeat.interval=5 sec nifi.cluster.protocol.is.secure=false # cluster node properties (only configure for cluster nodes) # nifi.cluster.is.node=true nifi.cluster.node.address=10.0.0.227 nifi.cluster.node.protocol.port=24427 nifi.cluster.node.protocol.threads=10 nifi.cluster.node.protocol.max.threads=50 nifi.cluster.node.event.history.size=25 nifi.cluster.node.connection.timeout=5 sec nifi.cluster.node.read.timeout=5 sec nifi.cluster.node.max.concurrent.requests=100 nifi.cluster.firewall.file= nifi.cluster.flow.election.max.wait.time=5 mins nifi.cluster.flow.election.max.candidates=3 # zookeeper properties, used for cluster management # nifi.zookeeper.connect.string=10.0.0.89:24489,10.0.0.227:24427,10.0.0.28:24428 nifi.zookeeper.connect.timeout=3 secs nifi.zookeeper.session.timeout=3 secs nifi.zookeeper.root.node=/nifi ##################################################################################10.0.0.228 Server####### ############################################################################## ####ZooKeeper.properties File##### clientPort=24428 initLimit=10 autopurge.purgeInterval=24 syncLimit=5 tickTime=2000 dataDir=./state/zookeeper autopurge.snapRetainCount=30 server.1=10.0.0.89:2888:3888 server.2=10.0.0.227:2888:3888 server.3=10.0.0.228:2888:3888 #####nifi.properties cluster section ########## # cluster common properties (all nodes must have same values) # nifi.cluster.protocol.heartbeat.interval=5 sec nifi.cluster.protocol.is.secure=false # cluster node properties (only configure for cluster nodes) # nifi.cluster.is.node=true nifi.cluster.node.address=10.0.0.228 nifi.cluster.node.protocol.port=24428 nifi.cluster.node.protocol.threads=10 nifi.cluster.node.protocol.max.threads=50 nifi.cluster.node.event.history.size=25 nifi.cluster.node.connection.timeout=5 sec nifi.cluster.node.read.timeout=5 sec nifi.cluster.node.max.concurrent.requests=100 nifi.cluster.firewall.file= nifi.cluster.flow.election.max.wait.time=5 mins nifi.cluster.flow.election.max.candidates=3 # zookeeper properties, used for cluster management # nifi.zookeeper.connect.string=10.0.0.89:24489,10.0.0.227:24427,10.0.0.28:24428 nifi.zookeeper.connect.timeout=3 secs nifi.zookeeper.session.timeout=3 secs nifi.zookeeper.root.node=/nifi ... View more

What would be the recommendation for a 40gbps NiFi cluster? And How would load balancing be handled with clustering? Would 1 IP be given for the cluster, lets say 1 HandleHTTP processor. and then the Network load will be distributed to the other nodes? Thank you! ... View more

I am unable to get TCP nifi to nifi to work. I have a put TCP processor that puts files to a port and host, but when the receiving host receives the TCP via listen TCP it receives a TON of small files that are not the original flowfile. Is listen TCP not putting the packets back together into a flowfile? or am I doing something wrong here? Attached are pics of my configs and setup. Thank you! ... View more

I have a few nifi boxes and they are load balanced using HAproxy. In our architecture we decided not to use NiFi Clustering. I have some data feeds that require In Order Delivery of files (xml). Since their data is load balanced across multiple servers, I cannot guarantee that the files will be delivered in the order that they were received. Any Ideas for how I could achieve this? Is there an attribute I could create? and a processor that maintains order? and waits for files of certain values to be in a queue before delivering them? Maybe a groovy script that could accomplish this (even an example of a groovy script that pays attention to attributes and multiple files would be helpful? John ... View more

In apache nifi how can I make the putfile processor use . rename as it writes a file out to the filesystem? or does it do this by default? Where it would write the file as a .filename and then rename it to the actual filename? John ... View more