Web service with Nifi - how handle when node where web service hosted goes down

[#1]

Hi,

I am exposing a webservice using HandleHTTPRequest on a Nifi cluster where webservice is exposed with hostname:port/service. My concern is if the node where webservice is hosted goes down, how can we configure or setup such that a the webserivce is hosted on the new node automatically.

I would like to know if there is configuration or solution for such auto deployment of webservice.

[#2]

@Anil Reddy

If I understand your question correctly, then you are looking for High Availability for your Web Service. Nifi is just a consumer pinging that web service. Assume any other client and not Nifi. How do you think you will manage this if client wasn't Nifi? I think the answer is load balancer for your web service. Or if your web service is working in a cluster then your cluster likely uses some web redirector that automatically switches over to another server if the server contacted is not available.

If I understand your question correctly, this is a server side problem and not client side (whether Nifi or any other client).

[#3]

@mqureshi

I am thinking in the lines as you said "if your web service is working in a cluster then your cluster likely uses some web redirector that automatically switches over to another server if the server contacted is not available.". But could not able to get the solution related to Nifi, hence posted here.

Small correction is, I am hosting webservice on Nifi which means Nifi is the server but not client.

[#4]

@Anil Reddy

The HandleHTTPRequest works in conjunction with the HandleHTTP Response processor.

In a NiFi cluster you should have this dataflow running on all nodes. If the node that is currently receiving the client connections goes down, the other nodes in the cluster will still be listening to receive data on the same port HandleHTTPRequest listening port. If the Node goes down before a HandleHTTPResponse message is returned, the client should timeout on waiting for a response and attempt to connect again. Any FlowFiles still queued in your dataflow beyond a successful HandleHTTPResponse has been send to the client will remain on that node until it is brought back online.

NiFi cannot control how the client handles a down/unreachable service. All NiFi can do is run a web service on every node in your cluster. Perhaps you client can auto switch to a different node's hostname/IP when it fails to get a response or connect to the current web service node?

Thanks,

Matt

[#5]

@Matt Clarke

thanks alot for your response. It gave lot of insights to how flow runs in Nifi cluster. I appreaciate your help.

Unfortunately, our requirement is a kind of registering service using the URL we host with webservice. So handling the change of hostname/ip on client side is not compatible solution in our scenario.

I would be interested to explore options how to handle exposing webservice with same URL as specified in the attachment.

Please let me know if you have exposure to such kind of architecture if it can be achieved with Nifi.

[#6]

@Anil Reddy

There are many ways to get data in and out of a NiFi instance/cluster.

Here is a good read from a friend of mine:

https://community.hortonworks.com/articles/16120/how-do-i-distribute-data-across-a-nifi-cluster.html

It mentions using a load balancer just as you described above.

Ignore the mention of "NCM" (NiFi Cluster Manager) as it only existed in Apache NiFi 0.x or HDF 1.x versions of NiFi. In newer version of Apache NiFi 1.x of HDF 2.x the NCM was eliminated.

Since you are likely using a newer version of NiFi, you can access the UI of any node at anytime to manage the cluster. Because of this you need to careful how you setup your load-balancer. It won't make difference when accessing the listening port of the HandleHTTPRequest, but it will impact access in the UI if you are not careful.

For example:

With a Secured NiFi instance that uses kerberos or LDAP for user authentication, when a user logs in to Node 1, a token is generated and provided to the user (stored in browser cache). This token is then passed in subsequent request against the NiFi end-point. This token is only good against the node that it was issued by. so if you load balancer directs the follow-on requests to a different node, they will be rejected. So you need to make sure a connected session continues to be routed to the same node.

Or if you only care about load balancing data delivery, don't put a load balancer in-front of NiFi's UI web ports. Only load-balance to the NiFi dataflow listener type processor's ports.

Thanks,

Matt

[#7]

@Anil Reddy

When you find an answer that answer your initial question, please mark it as accepted.

Thank you,

Matt