Community Articles

Find and share helpful community-sourced technical articles.
Labels (1)
avatar
Guru

Objective

With the release of NiFi 1.8.0, flowfiles that remain on a disconnected node can be rebalanced to other active nodes in the cluster via offloading.

Note: This tutorial assumes you have already setup a running NiFi cluster. The examples included are for a 2 node NiFi cluster.

Environment

This tutorial was tested using the following environment and components:

  • Mac OS X 10.11.6
  • Apache NiFi 1.8.0

Offload Nodes Tutorial

Queue FlowFiles

First, generate some queued flowfiles:

92996-1.png

In the Global Menu at the top left, select "Cluster" to see the Cluster Management dialog:

92997-2.png

The queued flowfiles are distributed between both nodes in the cluster:

92998-3.png

Disconnect Node

Select Disconnect for one of the nodes:

92999-4.png

The node is disconnected:

93000-5.png

Offload Node

Select Offload (93001-6.png) on the disconnected node. This will stop and terminate all processors and rebalance flowfiles to the other connected nodes in the cluster:

93002-7.png

Note: Offload would also stop transmitting on all remote process groups if they were in the flow.

When offloading is finished, all of the queued flowfiles are now on the active node:

93003-8.png

Delete & Decommission Node

An offloaded node can be either connected back to the cluster or deleted. Select Delete:

93004-9.png

Once deleted, the node cannot be rejoined to the cluster until it has been restarted. To decommission the node, stop/remove the NiFi service on the host.

6,629 Views
Comments
avatar
New Contributor

The tutorial was really informative. Can you please help me resolve the following doubts(Im a beginner in nifi).

 

1) Can we offload the flowfiles if the nifi process on that node is killed(or stopped) ? When I tried, the Status remains in OFFLOADING state instead of changing to OFFLOADED state.

 

2) Can we initiate the offloading process through curl script? as we do in the case of processors, to start and stop them?

 

3) Another doubt apart from the tutorial is, can we change the state of a processor if one node is disconnected from the cluster? When tried, it displays the following message :

"Cluster is unable to service request to change flow: Node is currently disconnected"

 

Please do mention if we can achieve this by any means.Thank you..

avatar
New Contributor

Can we offload the flowfiles if the nifi process on that node is killed(or stopped) ? When I tried, the Status remains in OFFLOADING state instead of changing to OFFLOADED state.