@Hans Feldmann - You are hitting a known bug in MiNiFi 0.4.0: https://issues.apache.org/jira/browse/MINIFI-435 - Fix is part of MiNiFi 0.5.0 or you can roll back to MiNiFi 0.3.0 which i understand works as well. - Thank you, Matt ... View more

@Davide Vergari @srinivas p - Let me explain what is going on here so you can understand why the configuration change made helps in some cases and not others. - When you add a component to the canvas of a NiFi cluster the following steps are performed. 1. That request is forwarded to the current elected cluster coordinator on behalf of the user who added the component. 2. The cluster coordinator then replicates that request to all the nodes connected in the cluster. (the nifi.cluster.node.protocol.threads=10 setting dictates how many concurrent request can be made, so larger clusters will need to have this value increased) 3. Each node must then make the change and respond back to the cluster coordinator. - While this process is consistent for every such replicated request, not all changes are equal. The action of selecting a bunch of components on the canvas copying and pasting them or instantiating a large template to the canvas also constitutes a single replication requests instead of many requests. These component bundles are referred to as snippets in NiFi. This action is not asynchronous. this means that each node must add every component (processors, connections, controller services, etc..) from this snippet before it responds to the cluster coordinator. Depending on size of the snippet and load on the server, this request may very likely exceed to configured timeout. This results in those nodes that timed out being disconnected from cluster. - Increasing the timeouts allows more time for these snippets to be instantiated and response to be received. Because there is no way to know how large these snippets are, the timeout setting that works for one user may not work for others. - Two things to keep in mind here: 1. How many node requests can be made concurrently (as shown above default is 10). Using a 16 node NiFi cluster as an example, 10 nodes must respond before the other 6 even get the request. so increase this value. 2. The nifi.cluster.node.connection.timeout and nifi.cluster.node.read.timeout can be set to much higher values. Even settings these timeouts to 2 - 5 minutes does not mean every request will take that long. It simply means that you will allow that much time before the cluster coordinator makes the decision to disconnect the node due to timeout. - There is work on the roadmap to redesign these types of replication requests in to asynchronous type requests eventually. Once that happens user will not need such high timeouts configured. - Thank you, Matt ... View more

@John T NiFi is a very difficult things to make a one size fits all sizing recommendation for. NiFi does not typically scale linearly. This is why you see the hardware specs exponentially increase as throughput increases. This is based on the fact that typical NiFi workflows all grow exponentially in size and complexity as the volume of throughput increases in most cases. More and more workflows are added. - Different NiFi processors in different workflows contribute to different server resource usage. That resource usage varies based processor configuration and FlowFile volume. So even two workflows using same processors may have different sizing needs. - How well a NiFi is going to perform has a lot to do with the workflow the user has build. After all it is this user designed work flow that is going to be using the majority of the resources on each node. - Best answer to be honest is to built your workflows and stress test them. This kind of a modeling and simulation setup. Learn the boundaries your workflows put on your hardware. At what data volume point does CPU utilization, network bandwidth, memory load, disk IO become my bottleneck for my specific workflow(s). Tweaking your workflows and component configurations. Then scale out by adding more nodes allowing some headroom considering it is very unlikely ever node will be processing the exact same number of NiFi FlowFiles all the time. - There are numerous way to handle load-balancing. It really depends on your dataflow design choices on how you intend to get data in to your NiFi. Keep in mind that each Nifi nodes in a cluster runs its own copy of the dataflows you build, has their own set of repositories, and thus works on their owns sets of FlowFiles. - While using NiFi's listener type processors would benefit from an external load-balancer to direct that incoming data across all nodes, processors like ConsumeKafka can run on all nodes consuming from same topic (assuming balanced number of Kafka partitions) - Other protocols like SFTP are not cluster friendly. So in dataflows like that you can only have something like ListenSFTP processor running on only one node at any given time. To achieve load-balancing there, a flow typically looks like: ListenSFTP (configured to run primary node only) ---> Remote Process Group (used to re-distribute/load-balance 0 byte FlowFiles to rest of nodes) --> input port --> FetchSFTP (Pulls content for each FlowFile). - One thing you do not want to do in most cases is load-balance the NiFi UI. You can do this but need to make sure you use sticky sessions in your load-balancer here. The tokens issued for user authentication (ldap or kerberos) are only good for node that issued them to user so subsequent requests must go to same node. - Hope this gives you some direction. - Thanks, Matt ... View more

@Raffaele S NiFi is a very difficult things to make a one size fits all sizing recommendation for. NiFi does not typically scale linearly. This is why you see the hardware specs exponentially increase as throughput increases. This is based on the fact that typical NiFi workflows all grow exponentially in size and complexity as the volume of throughput increases in most cases. More and more workflows are added. - Different NiFi processors in different workflows contribute to different server resource usage. That resource usage varies based processor configuration and FlowFile volume. So even two workflows using same processors may have different sizing needs. - How well a NiFi is going to perform has a lot to do with the workflow the user has build. After all it is this user designed work flow that is going to be using the majority of the resources on each node. - Best answer to be honest is to built your workflows and stress test them. This kind of a modeling and simulation setup. Learn the boundaries your workflows put on your hardware. At what data volume point does CPU utilization, network bandwidth, memory load, disk IO become my bottleneck for my specific workflow(s). Tweaking your workflows and component configurations. Then scale out by adding more nodes allowing some headroom considering it is very unlikely ever node will be processing the exact same number of NiFi FlowFiles all the time. - Thanks, Matt ... View more