Nifi Provenance, Flowfiles and Processor State persisting through docker container upgrade

[#1]

Hi, I'm running Nifi 1.1.2 on docker and I have some questions about preserving state. We're using ListSFTP and GetSFTP to get some files from a client of ours. At the moment we have nifi.database.directory, nifi.flowfile.repository.directory, nifi.provenance.repository.directory.default pointing to a docker volume that resides on shared storage. Our aim is to make sure that if we need to kill a docker host, or a container that we can pick up where we left off with a new container. At the moment this seems to be working with data provenance, but not with the state of ListSFTP or with the flowfiles themselves. Is there a supported method of doing this with nifi and docker? Or is there some other kind of mechanism that will let us preserve ListSFTP state, Flowfiles and Data Provenance through a reinstall of a nifi docker container?

[#2]

@Anthony Murphy

NiFi stores state differently depending on whether your NiFi is installed as cluster or a standalone instance.

With a cluster, NiFi stores state in zookeeper. As long as your new Nifi points to the same zookeeper or the zookeeper content has been moved to a new zookeeper you are using, state will be preserved.

In a Standalone Nifi install, state is recorded on disk.

You can look in your NiFi's state-management.xml file to see/change the configuration of both the "local-provider" (used by standalone NiFi) or "cluster-provider" (used by Clustered NiFi). You van change where state is being written to here.

Thanks,

Matt

[#3 · Accepted solution]

@Anthony Murphy

This should be fine if you have the appropriate volumes mapping to the specified directories. In this case, you should have three separate Docker volumes mapping your host-based shared location to the three directories in question. This will allow the Docker daemon to write to the external mappings and free it from the container. You would then invoke a new container with these same mappings and it will pick up where things left off.

If this is how you are attempting things, if you could please comment with your run command, we can certainly debug why things might be coming up short, but running a quick trial on my system, it looks like things are behaving as anticipated.

[#4]

I tried to run below command to mount a volume for the content repository. However, it always after a few minutes.

$ docker run -it --rm -v ~/nifi-content:/opt/nifi/nifi-1.2.0/content_repository -p 8080-8081:8080-8081 apache/nifi:1.2.0

It seems like file permission issue related with uid.

Would you please suggest how to make the docker container work?

Thanks.

[#5]

Is it possible to share your Docker Compose volume mappings for the three directories?

Trying this doesn't work when the container is removed then recreated:

nifi:

image: apache/nifi:latest

ports:

- 8443:8443

environment:

JAVA_OPTS: "-Xmx4g"

volumes:

- nifi-current:/opt/nifi/nifi-current

I'm trying to maintain templates and state in a standalone Nifi running in Docker Compose

[#6]

I have put detailed solution description here

You should be able to start NiFi & its registry with this docker compose file. It persists several folders on the local disk & rest will be persisted in the docker volume called 'nifi_data' so that it retains the previous state of other folders like content_repository, database_repository, flowfile_repository, provenance_repository, state  and work.