NiFi Cluster but single point of URL

[#1]

Hello All,

I have configured nifi flow with three node with external zookeeper

node1.domain.com

node2.domain.com

node3.domain.com

with three zookeeper nodes

zk1.domain.com

zk2.domain.com

zk3.domain.com

Everything is working fine but the requirement here is my flow has trigger points from external client systems to whom I have give a common URL like

node.domain.com to avoid any changes on the client teams if any of nodes goes down.

Any idea how to configure this.

[#2 · Accepted solution]

@Meeran 
The solution here is to setup an external load balancer in front of your 3 NiFi nodes.  Then have your clients point at that load balancer.  The Load Balancer would then be responsible for sending your client request to one of the available NiFi nodes. (If a node goes down, the Load Balancer simply does not send client requests to that node.

When using a Load Balancer in front of your NiFi cluster, it is important that the Load Balancer is configured to use sticky sessions.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

[#3 · Accepted solution]

@Meeran 
The solution here is to setup an external load balancer in front of your 3 NiFi nodes.  Then have your clients point at that load balancer.  The Load Balancer would then be responsible for sending your client request to one of the available NiFi nodes. (If a node goes down, the Load Balancer simply does not send client requests to that node.

When using a Load Balancer in front of your NiFi cluster, it is important that the Load Balancer is configured to use sticky sessions.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

[#4]

I have configured this and set the 
nifi.web.proxy.host: node1.domain.com:8443,node2.domain.com:8443,node3.domain.com:8443,loabalancerUrl.domain.com:8443

but while logging in I see below issue. I have checked the logs but no errors in nifi-user.logs

[#5]

Thanks for the reply @MattWho I have implemented in the same way what you have said but while accessing with the load balancer url I am getting the below exception

System Error
The request contained an invalid host header [internal-myappname-13718787520.us-east-1.elb.amazonaws.com:8443] in the request [/]. Check for request manipulation or third-party intercept.

[#6]

@Meeran Where are you seeing above exception?

System Error
The request contained an invalid host header [internal-myappname-13718787520.us-east-1.elb.amazonaws.com:8443] in the request [/]. Check for request manipulation or third-party intercept.

In your Load Balancer logs?
This host header "internal-myappname-13718787520.us-east-1.elb.amazonaws.com" is in your nifi.web.proxy.host list or in the SAN list within each of your node's certificates?

In NiFi, check the nifi-user.log and nifi-app.log for any WARN or ERROR log output being produced when you try to access NiFi via the load balancer URL.

[#7]

Hello @MattWho 

Load balancer started showing the login page after setting below properties in nifi.properties

nifi.web.proxy.context.path=/
nifi.web.proxy.host=myapp.dev.domain.com:8443  (load balancer url)

but after giving the logins details nifi canvas is not coming up

I am seeing the below error while access via load balancer URL in nifi-user.log file

2023-04-06 15:04:35,861 INFO [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 10.1xx.xx.xxx [<anonymous>] GET https://myapp.dev.domain.com:8443/nifi-api/flow/current-user
2023-04-06 15:04:35,862 WARN [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter

Authentication Failed 10.1xx.xx.xxx GET https://myapp.dev.domain.com:8443/nifi-api/flow/current-user [Anonymous authentication has not been configured.]

I am using LDAP for authentication.

Do I need to change any other properties/settings, please help

[#8 · Accepted solution]

@Meeran 

Did you setup sticky sessions in your load balancer?
When you provide your ldap username and password, a JWT Token issue to the client which is only valid for use with the specific NiFi node that issued it.  So if you enter your username and password and you LB sends next request to a different node to load the flow, the JWT token will not be accepted to identify your user on that other node.

Watch the nifi-user.log on all nodes when you attempt to login.  In one of those logs you will see authentication success logged for your user.  I suspect that then the above exception about  "anonymous" is then in a different node's user log.

If you are having trouble configuring your load balancer, it would be best to start a new community question for that query so it gets to a wider audience.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

[#9]

Thanks for support @MattWho . I overlooked that sticky session part but after configuring the sticky session on the load balancer everything is working fine.