Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi Load Balancing Demo

Solved Go to solution

NiFi Load Balancing Demo

Explorer

I'm trying to set up a Load Balancing Demo.

 

I have a GetFile processor that watcches a directory for incoming files and then a flow that processes them.

 

To try to understand LoadBalancing, I have created a second NiFi instance on the same host (my dev machine), but on a different port 8090 to the first 8080. I have then checked out the flow from the Registry so that it is identical on both instances.

 

Next, I have set the Load Balancing Strategy on the output queue from both nodes first processsor, (the GetFile proc) in order to see if I can get data going through both instances.

I can then drop files into the watch dir and see them hit my flow. However, all the files are going into the same node, even if I wait 10 minutes between dropping them into the dir. But, when I run again it switches to the other node and all the files go through there.

Is there a timeout somewhere that I can set to speed up the switch from one node to the other? Or better to switch files to alternate between the nodes.

I have assumed the use of the embedded Zookeepr and not made any specific cluster based config as yet...

Any help greatly appreciated. Note that this is all runnig on the same localhost, ports 8080 and 8090...

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: NiFi Load Balancing Demo

@JonnyL 

 

I would highly recommend that you back up and create a small 3 node nifi cluster to test this feature.  Putting 2 nifi on single node, does not satisfy the test cases you really want to be experimenting with.

 

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

 

Thanks,


Steven @ DFHZ

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

View solution in original post

2 REPLIES 2
Highlighted

Re: NiFi Load Balancing Demo

Expert Contributor

I have not tested it as such, but 10 minutes is a very long time for NiFi, I could imagine that the 'memory' of a load balancer is quite short to avoid overhead.

 

I would just drop in many files in a short period of time and see what happens. If you set the load balancer to round robin I would expect the files to go to both nodes.

 

Note that you can also use generateflowfile if you don't want to drop files all the time.


- Dennis Jaheruddin

If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'. Also check out my techincal portfolio at https://portfolio.jaheruddin.nl
Highlighted

Re: NiFi Load Balancing Demo

@JonnyL 

 

I would highly recommend that you back up and create a small 3 node nifi cluster to test this feature.  Putting 2 nifi on single node, does not satisfy the test cases you really want to be experimenting with.

 

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

 

Thanks,


Steven @ DFHZ

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here