Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Nifi Provenance, Flowfiles and Processor State persisting through docker container upgrade

avatar
Contributor

Hi, I'm running Nifi 1.1.2 on docker and I have some questions about preserving state. We're using ListSFTP and GetSFTP to get some files from a client of ours. At the moment we have nifi.database.directory, nifi.flowfile.repository.directory, nifi.provenance.repository.directory.default pointing to a docker volume that resides on shared storage. Our aim is to make sure that if we need to kill a docker host, or a container that we can pick up where we left off with a new container. At the moment this seems to be working with data provenance, but not with the state of ListSFTP or with the flowfiles themselves. Is there a supported method of doing this with nifi and docker? Or is there some other kind of mechanism that will let us preserve ListSFTP state, Flowfiles and Data Provenance through a reinstall of a nifi docker container?

1 ACCEPTED SOLUTION

avatar
Rising Star
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
5 REPLIES 5

avatar
Super Mentor

@Anthony Murphy

NiFi stores state differently depending on whether your NiFi is installed as cluster or a standalone instance.

With a cluster, NiFi stores state in zookeeper. As long as your new Nifi points to the same zookeeper or the zookeeper content has been moved to a new zookeeper you are using, state will be preserved.

In a Standalone Nifi install, state is recorded on disk.

You can look in your NiFi's state-management.xml file to see/change the configuration of both the "local-provider" (used by standalone NiFi) or "cluster-provider" (used by Clustered NiFi). You van change where state is being written to here.

Thanks,

Matt

avatar
Rising Star
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Expert Contributor

I tried to run below command to mount a volume for the content repository. However, it always after a few minutes.

$ docker run -it --rm -v ~/nifi-content:/opt/nifi/nifi-1.2.0/content_repository -p 8080-8081:8080-8081 apache/nifi:1.2.0

It seems like file permission issue related with uid.

Would you please suggest how to make the docker container work?

Thanks.

avatar
New Contributor

Is it possible to share your Docker Compose volume mappings for the three directories?

Trying this doesn't work when the container is removed then recreated:

 
nifi:
image: apache/nifi:latest
ports:
- 8443:8443
environment:
JAVA_OPTS: "-Xmx4g"
volumes:
- nifi-current:/opt/nifi/nifi-current
 
I'm trying to maintain templates and state in a standalone Nifi running in Docker Compose

avatar
Explorer

I have put detailed solution description here

 

You should be able to start NiFi & its registry with this docker compose file. It persists several folders on the local disk & rest will be persisted in the docker volume called 'nifi_data' so that it retains the previous state of other folders like content_repository, database_repository, flowfile_repository, provenance_repository, state  and work.