Support Questions

Find answers, ask questions, and share your expertise

Nifi Provenance, Flowfiles and Processor State persisting through docker container upgrade

avatar
Contributor

Hi, I'm running Nifi 1.1.2 on docker and I have some questions about preserving state. We're using ListSFTP and GetSFTP to get some files from a client of ours. At the moment we have nifi.database.directory, nifi.flowfile.repository.directory, nifi.provenance.repository.directory.default pointing to a docker volume that resides on shared storage. Our aim is to make sure that if we need to kill a docker host, or a container that we can pick up where we left off with a new container. At the moment this seems to be working with data provenance, but not with the state of ListSFTP or with the flowfiles themselves. Is there a supported method of doing this with nifi and docker? Or is there some other kind of mechanism that will let us preserve ListSFTP state, Flowfiles and Data Provenance through a reinstall of a nifi docker container?

1 ACCEPTED SOLUTION

avatar
Rising Star

@Anthony Murphy

This should be fine if you have the appropriate volumes mapping to the specified directories. In this case, you should have three separate Docker volumes mapping your host-based shared location to the three directories in question. This will allow the Docker daemon to write to the external mappings and free it from the container. You would then invoke a new container with these same mappings and it will pick up where things left off.

If this is how you are attempting things, if you could please comment with your run command, we can certainly debug why things might be coming up short, but running a quick trial on my system, it looks like things are behaving as anticipated.

View solution in original post

5 REPLIES 5

avatar
Master Mentor

@Anthony Murphy

NiFi stores state differently depending on whether your NiFi is installed as cluster or a standalone instance.

With a cluster, NiFi stores state in zookeeper. As long as your new Nifi points to the same zookeeper or the zookeeper content has been moved to a new zookeeper you are using, state will be preserved.

In a Standalone Nifi install, state is recorded on disk.

You can look in your NiFi's state-management.xml file to see/change the configuration of both the "local-provider" (used by standalone NiFi) or "cluster-provider" (used by Clustered NiFi). You van change where state is being written to here.

Thanks,

Matt

avatar
Rising Star

@Anthony Murphy

This should be fine if you have the appropriate volumes mapping to the specified directories. In this case, you should have three separate Docker volumes mapping your host-based shared location to the three directories in question. This will allow the Docker daemon to write to the external mappings and free it from the container. You would then invoke a new container with these same mappings and it will pick up where things left off.

If this is how you are attempting things, if you could please comment with your run command, we can certainly debug why things might be coming up short, but running a quick trial on my system, it looks like things are behaving as anticipated.

avatar
Expert Contributor

I tried to run below command to mount a volume for the content repository. However, it always after a few minutes.

$ docker run -it --rm -v ~/nifi-content:/opt/nifi/nifi-1.2.0/content_repository -p 8080-8081:8080-8081 apache/nifi:1.2.0

It seems like file permission issue related with uid.

Would you please suggest how to make the docker container work?

Thanks.

avatar
New Contributor

Is it possible to share your Docker Compose volume mappings for the three directories?

Trying this doesn't work when the container is removed then recreated:

 
nifi:
image: apache/nifi:latest
ports:
- 8443:8443
environment:
JAVA_OPTS: "-Xmx4g"
volumes:
- nifi-current:/opt/nifi/nifi-current
 
I'm trying to maintain templates and state in a standalone Nifi running in Docker Compose

avatar
Explorer

I have put detailed solution description here

 

You should be able to start NiFi & its registry with this docker compose file. It persists several folders on the local disk & rest will be persisted in the docker volume called 'nifi_data' so that it retains the previous state of other folders like content_repository, database_repository, flowfile_repository, provenance_repository, state  and work.