NiFi UI load balance with HAProxy to CONNECTED nodes only

[#1]

Hello, guys!

I still have a three-node cluster Apache NiFi 1.18.0 (yeap, upgrade task in by backlog =)) and trying to setup it behind HAProxy just for UI (/nifi endpoint).

I wanna to implement a health check to balance user requests only to nodes in CONNECTED status to avoid issues when nodes moves to maintenance (offload, disconnect).

Is it good idea, right? Or maybe I do some overengineering things? 😃

By the way, I have next HAProxy config file:

frontend nifi_443
  bind *:443 ssl crt /etc/haproxy/ssl/cert_file.pem
  http-request set-path /nifi if { path / }
  mode http
  option httplog
  acl DST_IP dst 10.0.0.10
  use_backend nifi_ui_9443 if DST_IP

backend nifi_ui_9443
  mode http
  option httpchk
  http-check send meth GET uri /nifi-api/access/config ver HTTP/1.1  hdr Host nifi-cluster.corp.company.com
  http-check expect status 200
  cookie SERVERID insert indirect nocache
  balance leastconn

  server srv_1 10.0.0.1:9443 check ssl verify none cookie web1
  server srv_2 10.0.0.2:9443 check ssl verify none cookie web2
  server srv_3 10.0.0.3:9443 check ssl verify none cookie web3

Basically it works fine when all nodes are CONNECTED, but if some node goes to DISCONNECTED state, http-check to selected API method still returns 200 and HAProxy continues to route users to that node.

So, give an advice, please. How can I configure HAProxy health-check to get expected behavior? Thanks a lot.

[#2 · Accepted solution]

@asand3r 

A node must be "disconnected" before it can be offloaded.  Only a running node can be offloaded.  So as long as a node is running. it's UI will be accessible.

A couple options:

1. https://<nifi-node>:<nifi-port>/nifi-api/controller/cluster. <-- this end point when hit will return the following type responses:
   • If node is connected, it will return json that includes the connection status of all nodes that are part of cluster.  So this response could be parsed to see if any nodes are disconnected. So you could parse status of all nodes from a single connected node's response
   • If node is not connected, it will return "Only a node connected to a cluster can process the request" which tells you that node is disconnected.
2. https://<nifinode>:<nifi-port>/nifi-api/flow/cluster/summary
   • If node is connected, it will return a response like this:
      {"clusterSummary":{"connectedNodes":"3 / 3","connectedNodeCount":3,"totalNodeCount":3,"connectedToCluster":true,"clustered":true}}
   • If node is disconnected, it will return a response like this:
     {"clusterSummary":{"connectedNodeCount":0,"totalNodeCount":0,"connectedToCluster":false,"clustered":true}}
   • You'll need to parse the responses for connectedToCluster. If true use that node's UI, if false, exclude that node's UI.
     
     
3. Of course if node is completely down, above will not help unless you mark as unavailable if no response.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3]

But there methods requires authentication. As I know it's not possible to create internal NiFi service user with persistent auth token to make a health check with it. So, right way is HAProxy agent check? 

[#4]

@asand3r 

Use a client certificate eliminates need for token when connecting with NiFi.
A secured NiFi will always want a client certificate first and only use other authentication methods when a client certificate is not presented in the TSL exchange.  This is how NiFi node perform authorized actions between nodes. 

[#5]

OK, @MattWho, thanks for your help.