Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Deployment in Production NiFi cluster

avatar

I am using 3 node NiFi (v1.4.0) cluster in production environment. I am in need of deploying new custom processors, but it requires NiFi restart, the challenge here is that NiFi receives chunks of data(s) per second. I would like to know the best possible way to deploy custom processors in NiFi cluster without interrupting the service.

1 ACCEPTED SOLUTION

avatar
Master Mentor

@Andrew thomas

When a NiFi node starts, it unpacks all its nar to the work directory. Before joining an existing cluster, it checks that the following three files locally match exactly what is being used in the cluster:

flow.xml.gz  <-- contains all configurations done via NiFi UI


users.xml  <-- Will only exist if NiFi is secured and the default NiFi file based authorizer is being used.


authorizations.xml  <-- Will only exist if NiFi is secured and the default NiFi file based authorizer is being used.

At no time does NiFi compare to make sure that all nodes are running the same set of nars. Mismatched nars only becomes an issue when that component is being used in the flow.xml.gz.

In your scenario where you are try to deploy a new custom nar which is not yet being used on your canvas you can update each node one at a time.

I recommend creating a new lib directory on each of your NiFi nodes that will hold all your custom nars separately from the default NiFi lib directory.

Then modify the nifi.properties file on each of your nodes (this can be done while NiFi is still running since this file is only read on startup). Add the following line to your nifi.properties files:

nifi.nar.library.directory.custom=/<path to my custom nifi nars directory>

Make sure this directory and your custom nars placed in this directory have proper ownership and permissions.

In order to minimize impact on your operational cluster, restart one node at a time. Once a restarted node has joined the cluster, move on to the next.

Once all nodes have been restarted and cluster has all nodes reconnected, you can start using your custom processor components on the canvas.

Thank you,

Matt

If you found the answer addressed your question, please take a moment to click on "accept" found below the answer.

View solution in original post

3 REPLIES 3

avatar
Master Mentor

@Andrew thomas

When a NiFi node starts, it unpacks all its nar to the work directory. Before joining an existing cluster, it checks that the following three files locally match exactly what is being used in the cluster:

flow.xml.gz  <-- contains all configurations done via NiFi UI


users.xml  <-- Will only exist if NiFi is secured and the default NiFi file based authorizer is being used.


authorizations.xml  <-- Will only exist if NiFi is secured and the default NiFi file based authorizer is being used.

At no time does NiFi compare to make sure that all nodes are running the same set of nars. Mismatched nars only becomes an issue when that component is being used in the flow.xml.gz.

In your scenario where you are try to deploy a new custom nar which is not yet being used on your canvas you can update each node one at a time.

I recommend creating a new lib directory on each of your NiFi nodes that will hold all your custom nars separately from the default NiFi lib directory.

Then modify the nifi.properties file on each of your nodes (this can be done while NiFi is still running since this file is only read on startup). Add the following line to your nifi.properties files:

nifi.nar.library.directory.custom=/<path to my custom nifi nars directory>

Make sure this directory and your custom nars placed in this directory have proper ownership and permissions.

In order to minimize impact on your operational cluster, restart one node at a time. Once a restarted node has joined the cluster, move on to the next.

Once all nodes have been restarted and cluster has all nodes reconnected, you can start using your custom processor components on the canvas.

Thank you,

Matt

If you found the answer addressed your question, please take a moment to click on "accept" found below the answer.

avatar

@Matt Clarke Thanks for the clear explanation on NiFi custom processor deployment. Also, I would like to know if we can deploy NiFi flow and configuration node by node without interrupting the service.

avatar
Master Mentor

@Andrew thomas

Once you have a NiFi cluster installed and running, any change you make within the NiFi Ui affects every node in the cluster. There is no way in a NiFi cluster to deploy a flow to just one node only.

There is no need to stop any existing running flow within the UI when you are adding/building new flows within the same UI.

Any configuration changes that affect any one of NiFi's configuration files (except logback.xml) will require a NiFi restart before those changes will take affect.

Thank you,

Matt