NiFi Docker image with inbuilt .xml templates under /opt/nifi/conf/templates

[#1]

I am creating a NiFi Docker image wherein in am copying existing xml templates under /opt/nifi/conf/templates When i start this Docker container with following command it starts the instance and display all existing templates.

docker run -d --name nifi -p 7777-7778:7777-7778 nifia But when i execute docker stop nifi and docker start nifi , it starts the container but it does not display the UI and container automatically stops after 5-7 minutes. Find attached file containing my dockerfile and templates.

Please help to sort out this mistery.

nifi.tar.gz

[#2]

Dear Experts, Is it fine to copy existing templates under /opt/nifi/conf/templates though newly created templates does not get stored under /opt/nifi/conf/templates which could be the reason why my nifi docker doesnt start up after docker stop nifi. Rerunning docker from image will delete my newly created templates. docker run -d --name nifi -p 7777-7778:7777-7778 nifia Am i following the right approach? I have attached my Dockerfile and run.py in above post. Please have a quick look and suggest to proceed further.

[#3 · Accepted solution]

Keep in mind that a Docker image is immutable and when any changes are made, this is both a new layer and image. To save the templates in your current approach, you would need to save the Docker image via a docker commit. Although this would not be maintainable, nor would it be best practice.

Instead, you would likely want to make use of Docker volumes (https://docs.docker.com/engine/tutorials/dockervolumes/) such that the data could be persistent on the host. In this case, you could use a folder on the host that contains your seed templates that would be mapped into running instances of the image you created. Something to the effect of:

docker run -it -v /<host templates dir>/:/opt/nifi/conf/templates/ my_image

This will allow you to provide templates to your container and also capture those new ones introduced in the image on the host system. This set of cached templates on your host will also enable you to then mount these templates to multiple instances that would be colocated on that host.

[#4]

@Aldrin Piri Thanks for your comment! i got your point to put these templates under host templates dir and mount that as docker volume. But i would need to create new instances on different hosts. That mean i will have to copy/paste templates to all hosts. Instead it would be better to keep templates within docker image and add them to /opt/nifi/conf/templates during image build.

Can you please help to suggest me in this direction.

1. new docker instance should have some previously created templates. Say abc.xml

2. create new templates in nifi instance say xyz.xml (dont know the default storage location of newly created template.)

3. docker stop nifi

4. docker start nifi (Should come up with abc.xml & xyz.xml templates.)

[#5]

Yes, this dynamic will not work in the current scenario. There is some work under way and has been proposed to help in these scenarios. You can read about that here: https://cwiki.apache.org/confluence/display/NIFI/Configuration+Management+of+Flows

As mentioned prior, the only way to accomplish what you are looking for is to perform a docker commit and use that image as the new point of reference any time you want to capture the current state of the instance but would include the totality of the running instance. There are some alternative storage drivers that allow volumes to be shared which are also covered at the previously linked Docker documentation, such as Flocker. https://docs.docker.com/engine/tutorials/dockervolumes/#/mount-a-shared-storage-volume-as-a-data-vol...

[#6]

Hi Aldrin, Thanks for your response! I understood that there is some work under way considering this path.

In the mean time, Could you please help me with NiFi APIs/Commands to export xml templates from one instance and import them to second instance .

I can add them to my docker file so as to store exported templates at default location which is different from /opt/nifi/conf/templates.

[#7]

Hi, I have a poc kubernetes setup at

https://github.com/whs-dot-hk/kubernetes-nifi-refined

which also works for docker and docker compose