Member since
10-08-2016
59
Posts
16
Kudos Received
0
Solutions
06-11-2018
01:22 AM
@Matt Burgess any ideas as well?
... View more
06-11-2018
01:21 AM
@Matt Clarke Any ideas?
... View more
06-10-2018
11:35 PM
1 Kudo
So I have 4 740xd Servers with 80 Threads on each and 384GB of ram each.
How should I best handle heap for each nifi VM on the server?
Option 1: 1, VM 80 cpus, NiFi set to 300GB of Ram for heap? And have the 4 nifis clustered.
Option 2: 10, VMs 8 cpus each, NiFi set to 32GB of Ram for heap for each VM, These NiFis would also be clustered, across the Virtualized 740 and across the 4 hardware 740s
Note: all these files going through these nifis are less than 4GB, and 90% of them are under 10MB
ALSO what generation should I use for Java Garbage Collection?
Attached are two pictures explaining my options.
John
... View more
Labels:
- Labels:
-
Apache NiFi
05-07-2018
04:53 PM
Matt, Thank you for your thorough feedback! How does nifi handle load balancing on its own if there was 1 listenHTTP processor on a cluster? Would we give the endpoint communicating customer 1 IP to reach us? And then nifi would broker that connection to one of the nodes in the cluster? Or would the data come through 1 node and then distribute to the other nodes? Basically I'm trying to use nifi clustering as the load balancer 🙂 Any suggestions? If that doesen't work I was going to try out HAproxy in front of the cluster IP. John
... View more
05-07-2018
03:41 PM
What would be the recommendation for a 40gbps NiFi cluster?
And
How would load balancing be handled with clustering? Would 1 IP be given for the cluster, lets say 1 HandleHTTP processor. and then the Network load will be distributed to the other nodes?
Thank you!
... View more
Labels:
- Labels:
-
Apache NiFi
02-05-2018
10:45 PM
@Matt Clarke Hey Matt, Thanks for the quick feedback! So I can just change the batch size on the listen tcp to a really large number and regaurdless of the actual number of new lines it will package back together a full flowfile? I really need to do TCP to TCP because there is a middle system I will have to send my stream through, that will inspect the stream and TCP is the only way to do so. Any other ideas? John
... View more
02-05-2018
03:29 PM
1 Kudo
I am unable to get TCP nifi to nifi to work.
I have a put TCP processor that puts files to a port and host, but when the receiving host receives the TCP via listen TCP it receives a TON of small files that are not the original flowfile.
Is listen TCP not putting the packets back together into a flowfile? or am I doing something wrong here?
Attached are pics of my configs and setup.
Thank you!
... View more
Labels:
- Labels:
-
Apache NiFi
03-13-2017
04:30 PM
Thank you!! That worked perfectly! Feel free to check out my NiFi template where this is in use! original-content-and-attributes-in-xmlv4.xml
... View more