NiFi templates don't include all controller services

[#1]

Hi all,

I have a NiFi flow using DistributedMapCacheServer 1.11.3. I've noticed that this controller service is actually excluded when I create a template from the process group. I see other controller services that are being referenced from the flow in the template, but not this one, presumably because it isn't referenced.

How exactly do I get this exported to a template? Any ideas?

Regards

[#2]

Hi ,

If Controller Services created at base processor group level , then you will not get controller services when you create a template only for the subprocessor group . You will have to create a template for base processor group of immediate child processor group or define the distributed mapcache controller service in the processor group which you would want to create a template.

For example 'Stream Ingestion' is immediate super processorgroup of 'TestSample' processorgorup . If i create a template for 'TestSample' processor group then in the template i can not see the DistributedMapCacheServer controller service because the specific controller service scope is in Streaming Ingestion processgroup as a template.

Controller Services created within a Process Group will be available/referenced to all descendant components.

Thanks

[#3]

Hi, thanks for the response!

What I have is a process group called "test", and inside is the controller service "DistributedMapCacheServer" + some other processors.

When I create the template of "test", I get the processors and controller services in the template, except for DistributedMapCacheServer. I have a feeling it's because DistributedMapCacheServer is not using a SSL context service and it has no connections to any processors/controller services, so it's being excluded. Does that make sense?

The cache client is include (it's referenced by a processor so it shows up under relationships) but the cache server is still not included.

[#4 · Accepted solution]

I just did a small test and it indeed seems that only referenced controller services are copied when you copy a process group.

I have not found any discussion on the topic, but as it seems to happen for all (just tried csvrecordreader) I suspect it is a feature. I hope this does not seem too unreasonable.

[#5]

That's unfortunate, I'd like to be able to create a template from everything in the process group.  Thanks for your help.

[#6]

@jg6 

There is no direct relationship between the DistributedMapCacheServer and the DistributedMapCacheClientService.  Meaning that the client is simply configured with a hostname and a port.  This hostname and port could be a DistributedMapCacheServer running on an entirely different NiFi cluster somewhere.

Additionally there is no component that registers a dependency on the DistributedMapCacheServer controller service.  They only have a dependency on the DistributedMapCacheClientService.  So when constructing a template only the interconnected and dependent pieces are included.

That being said, using the DistributedMapCache is not the cache I would recommend using anyway.  IT offers no high Availability (HA).  While a DistributedMapCacheServer is being started on every node in a NiFi cluster, they do not talk to one another and the DistributedMapCacheClientService can only be configured to point at one of them.  So if you lose the NiFi node were your clients point, you lost all your cache.  There are better options for external cache services that do offer HA.

Hope this is helpful,

Matt