Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi Provenance, Flowfiles and Processor State persisting through docker container upgrade

Solved Go to solution

Nifi Provenance, Flowfiles and Processor State persisting through docker container upgrade

New Contributor

Hi, I'm running Nifi 1.1.2 on docker and I have some questions about preserving state. We're using ListSFTP and GetSFTP to get some files from a client of ours. At the moment we have nifi.database.directory, nifi.flowfile.repository.directory, nifi.provenance.repository.directory.default pointing to a docker volume that resides on shared storage. Our aim is to make sure that if we need to kill a docker host, or a container that we can pick up where we left off with a new container. At the moment this seems to be working with data provenance, but not with the state of ListSFTP or with the flowfiles themselves. Is there a supported method of doing this with nifi and docker? Or is there some other kind of mechanism that will let us preserve ListSFTP state, Flowfiles and Data Provenance through a reinstall of a nifi docker container?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Nifi Provenance, Flowfiles and Processor State persisting through docker container upgrade

Contributor

@Anthony Murphy

This should be fine if you have the appropriate volumes mapping to the specified directories. In this case, you should have three separate Docker volumes mapping your host-based shared location to the three directories in question. This will allow the Docker daemon to write to the external mappings and free it from the container. You would then invoke a new container with these same mappings and it will pick up where things left off.

If this is how you are attempting things, if you could please comment with your run command, we can certainly debug why things might be coming up short, but running a quick trial on my system, it looks like things are behaving as anticipated.

4 REPLIES 4

Re: Nifi Provenance, Flowfiles and Processor State persisting through docker container upgrade

Master Guru

@Anthony Murphy

NiFi stores state differently depending on whether your NiFi is installed as cluster or a standalone instance.

With a cluster, NiFi stores state in zookeeper. As long as your new Nifi points to the same zookeeper or the zookeeper content has been moved to a new zookeeper you are using, state will be preserved.

In a Standalone Nifi install, state is recorded on disk.

You can look in your NiFi's state-management.xml file to see/change the configuration of both the "local-provider" (used by standalone NiFi) or "cluster-provider" (used by Clustered NiFi). You van change where state is being written to here.

Thanks,

Matt

Re: Nifi Provenance, Flowfiles and Processor State persisting through docker container upgrade

Contributor

@Anthony Murphy

This should be fine if you have the appropriate volumes mapping to the specified directories. In this case, you should have three separate Docker volumes mapping your host-based shared location to the three directories in question. This will allow the Docker daemon to write to the external mappings and free it from the container. You would then invoke a new container with these same mappings and it will pick up where things left off.

If this is how you are attempting things, if you could please comment with your run command, we can certainly debug why things might be coming up short, but running a quick trial on my system, it looks like things are behaving as anticipated.

Re: Nifi Provenance, Flowfiles and Processor State persisting through docker container upgrade

Contributor

I tried to run below command to mount a volume for the content repository. However, it always after a few minutes.

$ docker run -it --rm -v ~/nifi-content:/opt/nifi/nifi-1.2.0/content_repository -p 8080-8081:8080-8081 apache/nifi:1.2.0

It seems like file permission issue related with uid.

Would you please suggest how to make the docker container work?

Thanks.

Highlighted

Re: Nifi Provenance, Flowfiles and Processor State persisting through docker container upgrade

New Contributor

Is it possible to share your Docker Compose volume mappings for the three directories?

Trying this doesn't work when the container is removed then recreated:

 
nifi:
image: apache/nifi:latest
ports:
- 8443:8443
environment:
JAVA_OPTS: "-Xmx4g"
volumes:
- nifi-current:/opt/nifi/nifi-current
 
I'm trying to maintain templates and state in a standalone Nifi running in Docker Compose
Don't have an account?
Coming from Hortonworks? Activate your account here