About mike-nacey

mike-nacey · ‎07-30-2018

So far I have been able to get this working. Traffic flows fine through the final NLB, but we want to do some better load testing. I have put together a post that explains: https://everymansravings.wordpress.com/2018/07/27/apache-nifi-behind-an-aws-load-balancer-w-minifi/

mike-nacey · ‎07-25-2018

We are using the standard Java minifi agent to query a database and send changed records out to nifi over S2S (frequent small updates). The main reason for the LB is not necessarily to balance the S2S load. It is to: 1. Obfuscate knowledge of the individual NiFi nodes from the minifi agent (networking best practice). 2. Make sure that the S2S host specified in the minifi config is not a single point of failure (meaning if that node happens to be down). 3. Keep the NiFi nodes in a private subnet (private security zone) and expose only the minimum required in a public zone. In addition, all traffic comes in through a "transit VPC", meaning it has passed through (before reaching NiFi): 1. An edge balancer (balancing firewalls) 2. A firewall 3. A second balancer (balancing WAFs) 4. A WAF/RP that inspects traffic and routes accordingly, as well as inspecting traffic for funny business It's mostly like this: https://www.draw.io/#LNifi.xml

mike-nacey · ‎07-25-2018

I think you are better off continuing to use hostnames to make this work. SSL is really not designed to be used for IP addresses. In production, you would want to use a DNS name for this purpose anyhow.

mike-nacey · ‎07-25-2018

I see there are some settings added for S2S use through Reverse Proxy. I will try those out.

mike-nacey · ‎07-24-2018

So I think you are saying that NiFi will direct S2S around the load balancer (i.e. to a specific node's internal IP) being used to abstract the individual nodes, similar to the behavior of Kafka? This is problematic. Single node migration is probably not a good option since this is being managed through HDF. How do I abstract the NiFI cluster members from the 800 MiniFi edge nodes that can almost never be updated?

mike-nacey · ‎07-23-2018

Thanks. What I have found is that incoming requests from the LB (aws NLB) were being rejected, even when the LB DNS was part of the SAN in the cert. Setting the nifi.web.proxy.host = <LB DNS>:9091 stopped this from happening, but now there is an error saying that the hostname should be <X>, where X is the private DNS name of the node. I will try adding the private DNS entries to the SAN to see if this resolves. Thanks.

mike-nacey · ‎07-23-2018

Does anyone have recommendations about how to use MiniFi with NiFi in a real production environment? For instance, I need to deploy MiniFi on a number of edge nodes outside of my network (customer data centers). I want to point MiniFi to my NiFi cluster in AWS. I DO NOT want my MiniFi to point at the individual NiFi cluster members (that would be poor network design) since I may change out NiFi clusters (Blue/Green). Changing the MiniFi configurations on the edge nodes should be done almost never. What I would like to do is to to add a load balancer that would balance (MiniFi) traffic to the NiFi nodes, not so much for the load balancing as to hide the NiFi node DNS's from MiniFi (like you do). That way when I switch out a NiFi cluster (let's say to update NiFi version), I can just point the DNS that was pointing to my original load balancer to a new load balancer for the new cluster. The client cert/SSL design of NiFi makes this difficult, and the recently added host header checking (1.5) even more difficult. Does anyone have this working properly? Thanks

mike-nacey · ‎07-23-2018

Is there a way to turn off the Header Request checking entirely? This is causing hoopla with load balancers. SSL will take care of this on its own, yes? , Is there a way to disable the Request Header check? This causes hellavu problems with load balancers.

Online	Offline
Last Visited	‎07-30-2018 09:06 PM

Member Since	‎07-23-2018 06:49 PM
Last Visited	‎07-30-2018 09:06 PM
Posts	8
Kudos received	1

Cloudera Community

Cloudera Community

Re: MiniFi to NiFi connection through load balance...

Re: MiniFi to NiFi connection through load balance...

Re: Secure Remote Process Group communication with...

Re: MiniFi to NiFi connection through load balance...

Re: MiniFi to NiFi connection through load balance...

Re: NiFi 1.5 System Error: Invalid host header

MiniFi to NiFi connection through load balancer?

Re: NiFi 1.5 System Error: Invalid host header