Created 05-09-2017 11:51 PM
Hi, I'm running Nifi 1.1.2 on docker and I have some questions about preserving state. We're using ListSFTP and GetSFTP to get some files from a client of ours. At the moment we have nifi.database.directory, nifi.flowfile.repository.directory, nifi.provenance.repository.directory.default pointing to a docker volume that resides on shared storage. Our aim is to make sure that if we need to kill a docker host, or a container that we can pick up where we left off with a new container. At the moment this seems to be working with data provenance, but not with the state of ListSFTP or with the flowfiles themselves. Is there a supported method of doing this with nifi and docker? Or is there some other kind of mechanism that will let us preserve ListSFTP state, Flowfiles and Data Provenance through a reinstall of a nifi docker container?
Created 05-17-2017 02:52 PM
This should be fine if you have the appropriate volumes mapping to the specified directories. In this case, you should have three separate Docker volumes mapping your host-based shared location to the three directories in question. This will allow the Docker daemon to write to the external mappings and free it from the container. You would then invoke a new container with these same mappings and it will pick up where things left off.
If this is how you are attempting things, if you could please comment with your run command, we can certainly debug why things might be coming up short, but running a quick trial on my system, it looks like things are behaving as anticipated.
Created 05-10-2017 12:44 PM
NiFi stores state differently depending on whether your NiFi is installed as cluster or a standalone instance.
With a cluster, NiFi stores state in zookeeper. As long as your new Nifi points to the same zookeeper or the zookeeper content has been moved to a new zookeeper you are using, state will be preserved.
In a Standalone Nifi install, state is recorded on disk.
You can look in your NiFi's state-management.xml file to see/change the configuration of both the "local-provider" (used by standalone NiFi) or "cluster-provider" (used by Clustered NiFi). You van change where state is being written to here.
Thanks,
Matt
Created 05-17-2017 02:52 PM
This should be fine if you have the appropriate volumes mapping to the specified directories. In this case, you should have three separate Docker volumes mapping your host-based shared location to the three directories in question. This will allow the Docker daemon to write to the external mappings and free it from the container. You would then invoke a new container with these same mappings and it will pick up where things left off.
If this is how you are attempting things, if you could please comment with your run command, we can certainly debug why things might be coming up short, but running a quick trial on my system, it looks like things are behaving as anticipated.
Created 05-29-2017 07:20 PM
I tried to run below command to mount a volume for the content repository. However, it always after a few minutes.
$ docker run -it --rm -v ~/nifi-content:/opt/nifi/nifi-1.2.0/content_repository -p 8080-8081:8080-8081 apache/nifi:1.2.0
It seems like file permission issue related with uid.
Would you please suggest how to make the docker container work?
Thanks.
Created 09-05-2019 05:47 AM
Is it possible to share your Docker Compose volume mappings for the three directories?
Trying this doesn't work when the container is removed then recreated:
Created 09-14-2020 12:49 AM
I have put detailed solution description here
You should be able to start NiFi & its registry with this docker compose file. It persists several folders on the local disk & rest will be persisted in the docker volume called 'nifi_data' so that it retains the previous state of other folders like content_repository, database_repository, flowfile_repository, provenance_repository, state and work.