NiFi Load Balancing Demo

Question

I'm trying to set up a Load Balancing Demo.

I have a GetFile processor that watcches a directory for incoming files and then a flow that processes them.

To try to understand LoadBalancing, I have created a second NiFi instance on the same host (my dev machine), but on a different port 8090 to the first 8080. I have then checked out the flow from the Registry so that it is identical on both instances.

Next, I have set the Load Balancing Strategy on the output queue from both nodes first processsor, (the GetFile proc) in order to see if I can get data going through both instances.

I can then drop files into the watch dir and see them hit my flow. However, all the files are going into the same node, even if I wait 10 minutes between dropping them into the dir. But, when I run again it switches to the other node and all the files go through there.

Is there a timeout somewhere that I can set to speed up the switch from one node to the other? Or better to switch files to alternate between the nodes.

I have assumed the use of the embedded Zookeepr and not made any specific cluster based config as yet...

Any help greatly appreciated. Note that this is all runnig on the same localhost, ports 8080 and 8090...

stevenmatison · Accepted Answer

@JonnyL

I would highly recommend that you back up and create a small 3 node nifi cluster to test this feature. Putting 2 nifi on single node, does not satisfy the test cases you really want to be experimenting with.

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.

Thanks,

Steven @ DFHZ

DennisJaheruddi · Answer

I have not tested it as such, but 10 minutes is a very long time for NiFi, I could imagine that the 'memory' of a load balancer is quite short to avoid overhead.

I would just drop in many files in a short period of time and see what happens. If you set the load balancer to round robin I would expect the files to go to both nodes.

Note that you can also use generateflowfile if you don't want to drop files all the time.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.