How to upgrade NiFi Cluster in place?

[#1]

Hi guys,

I was wondering what's the best practice to upgrade NiFi cluster version, e.g.from 1.0 to 1.3

1. Is there a patch to upgrade the cluster incrementally in live?

2. Should we have another trunk with new version, then switch business from old trunk to new trunk?

If we do this way, how to migrate the data in the repositories?

3. Can we upgrade in the existing cluster nodes? Shut down all the NiFi nodes, then deploy the new NiFi version, and start them.

In this case, how do we handle the business downtime during upgrading, for real-time streaming data.

How to handle the data still in workflow before upgrading?

4. Could we upgrade NiFi node one by one? shut down only one node and upgrade it at once. I guess we will have incompatible issue.

Thanks.

[#2 · Accepted solution]

Currently you have to shutdown all the nodes, upgrade them, and restart. There may be future improvements to allow a rolling upgrade (node-by-node), but currently you can't have live nodes with mixed versions, meaning node1 is upgrade and comes up with 1.3 and the rest of the cluster is still running 1.0.

From an upgrade perspective, you are mostly replacing the lib directory with the lib directory from the latest release, and of course comparing config files to see what has changed, and reading the release notes, but how you do this is really up to you. There is an article here with some tips:

https://cwiki.apache.org/confluence/display/NIFI/Upgrading+NiFi

Any data in your flow will still be there when you restart.

[#3]

HDP and HDF are packaged distributions and are the units of software supported by Hortonworks - upgrading individual components of the stack is not currently supported.

To perform an HDF upgrade you can do an express upgrade through Ambari - this is the supported method of upgrading the HDF distribution. This will shutdown your entire cluster, upgrade the software, and then bring the cluster back online.

Ambari upgrade documentation: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.0/bk_ambari-upgrade/content/preparing-to-upgr...

To answer your other questions:

-Real-time streaming data will have to be queued/buffered elsewhere upstream as you are doing the upgrade, there will be some downtime as Ambari upgrades the HDF stack. This is a wider architectural configuration that must be accounted for.

-As far as how to handle data still in the workflow - you can turn off the first processor in the flow so that all downstream processors can finish what they are working on. There will be no data in the flow then, and you can safely turn off all processors and start the upgrade.

-The rolling upgrade (upgrading nodes one-by-one) is not currently supported for the HDF distribution, it is however supported for HDP. We are actively looking into how to make rolling upgrades work for HDF.

[#4]

Thank you @anarasimham

You made a good point on "turn off the first processor in the flow before turning off the whole flow"