Apache Nifi 1.23.2 FetchDistributedMapCache Connection Refused

[#1]

I started DistributedMapCacheClientService on Controller Services.when i try to start FetchDistributedMapCache processor got an error "AnnotatedConnectException: Connection refused".Do you have any comments on this situation?

@MattWho 

[#2]

@plapla 
I see that you have configured your DistributedMapCacheClient controller service with a StandardSSLContextService.  You did not share your configuration for the needed DistributedMapCachServer.  I must assume that the server was also configured with a StandardRestrictedSSLContextService.

So few things to comment on here related to DistributedMapCache:
1. You configured the client with "localhost". Does localhost exist as a SAN entry in the PrivateKeyEntry found in the Keystore used in the DistributedMapCache server's configured keystore within the StandardRestrictedSSLContextService?
2. Does the keystore PrivateKeyEntry used in the StandardSSLContext services used by the DistributedMapCacheClient service include ClientAuth as an Extended Key Usage (EKU)?
3. Using "localhost means that all nodes will be using different map caches.  The DistributedMapCacheServer starts up a separate Distributed Map Cache server on each host in the NiFi cluster. They do not communicate with one another to share cached data.  In a cluster using this specific embedded cache server it is not common to use "localhost", but rather configured the hostname of one specific host in your NiFi cluster.
4. Once you had your DistributedMapCache server setup and enabled, Did you then start your DistributedMapCacheClient service and click the validation test?
5. When you started the DistributedMapCacheServer, did any of the NiFi cluster nodes throw any exceptions in the nifi-app.log related to possible port 4557 already in use?

Without the verbose output of your Keystore and truststore files (all 4 ) used by both the DistributedMapCache server and client, it would be impossible for me to tell you if they are the issue here or not.  But... I suspect you may have a MutualTLS exchange issue here.


History: The DistributedMapCache (Server and Client) controller services have exists for almost as long as NiFi has existed.  They were originally implemented before NiFi offered its zero master cluster capability.  Very old versions o NiFi had a NiFi Cluster Manager (NCM) and then worker nodes that connected to that NCM.  At that time the DistributedMapCacheServer would only be installed and run on that NCM.  So DMCacheClient would always be configured to the one NCM host.  Now with Zero Master clustering there is not NCM and thus the server start a map cache server on all nodes. It is for this reason that more and newer DistributedMapCache client options are available that offer HA which is not offered through the DistributedMapCache.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3]

Hi @MattWho ,

First of all, thank you for your return.

• I tried it three ways for testing.Firstly DistributedMapCacheClientService parameter ssl context service set  no value.Since it is not successful this way I try to StandardSSLContextService and StandardRestrictedSSLContextService.
•  Nifi process running on single server.Nifi security parameters are set as follows.I added the values ​​in the nifi.properties file in StandartSSLContextService.Should I use SSL since NiFi is secure?Or is my method wrong?

nifi.security.keystore=/opt/nifi-1.23.2/certs/keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=XXXX
nifi.security.keyPasswd=XXXX
nifi.security.truststore=/opt/nifi-1.23.2/certs/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=XXXX

• When I enable to StandardSSLContextService,the logs like this.

2023-10-31 12:24:49,269 INFO [Timer-Driven Process Thread-6] o.a.n.c.s.StandardControllerServiceNode Successfully enabled StandardControllerServiceNode[service=SSLContextService[id=6c5e44ef-018b-1000-19c7-5f91c320527d], name=StandardSSLContextService, active=true]

• After enable ssl context ,enable DistributedMapCacheClientService.Logs lik this.

2023-10-31 12:30:14,909 INFO [Timer-Driven Process Thread-4] o.a.n.c.s.StandardControllerServiceNode Successfully enabled StandardControllerServiceNode[service=DistributedMapCacheClientService[id=6c484bbd-018b-1000-a778-92a07c63c0a2], name= DistributedMapCacheClientService, active=true]

• Finally, when I start the processor, I get an error like the one below.When I try it with different ports the result is the same.Telnet commands give same connection refused error.

2023-10-31 12:43:10,977 ERROR [Timer-Driven Process Thread-5] o.a.n.p.s.FetchDistributedMapCache FetchDistributedMapCache[id=3d4b463d-3954-3fec-35b1-c50417c0d8da] Unable to communicate with cache when processing StandardFlowFileRecord[uuid=b1c817fa-cc08-41f0-a3f9-6327ebb0ba74,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1698324770398-90, container=default, section=90], offset=11595, length=184],offset=0,name=b1c817fa-cc08-41f0-a3f9-6327ebb0ba74,size=184] due to io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost:4557
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost:4557
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)

[#4 · Accepted solution]

@plapla 
The DistributedMapCacheClientService is used to connect to the DistributedMapCacheServer.  You have not shared how you have the DistributedMapCacheServer controller service configured.

The DistributedMapCacheServer configuration determines whether the DistributedMapCacheClientService needs tobe configured with a StandardRestrictedSSLContext Service.  Essentially if you configure a StandardRestrictedSSLContext service on the DistributedMapCacheServer, then it must also be configured on the DistributedMapCacheClientService.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt