Error Securing NiFi Cluster with a Single Certificate Across 3 Nodes

[#1]

I have a NiFi cluster consisting of 3 nodes, and I secured the cluster using a single signed certificate for all nodes. However, I am encountering an error that I suspect might be due to using just one certificate.

Error Details:

- Logs: 

[Replicate Request Thread-25] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/current-user to  nifi01:8443 due to javax.net.ssl.SSLPeerUnverifiedException: Hostname nifi01 not verified:

    certificate: sha256/*********/GessD8=

    DN: CN=nifi01

    subjectAltNames: [nifi03,nifi02]

2024-06-13 17:34:07,555 WARN [Replicate Request Thread-25] o.a.n.c.c.h.r.ThreadPoolRequestReplicator

javax.net.ssl.SSLPeerUnverifiedException: Hostname nifi01 not verified:

    certificate: sha256/************/GessD8=

    DN: CN=nifi01

    subjectAltNames: [nifi03,nifi02]

        at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.kt:389)

        at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.kt:337)

        at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:209)

        at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)

        at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)

        at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)

        at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)

        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)

        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)

        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)

        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)

        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)

        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)

        at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)

        at org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:136)

        at org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:130)

        at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:645)

        at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:869)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:750)

Could using a single certificate for all three nodes(imported in truststore of all nodes) be causing this issue? Any guidance or best practices would be greatly appreciated.

[#2 · Accepted solution]

Hi,

Usually when you get this error it means the certificate is not setup correctly to work with Nifi. For example if you are using wildcard certificate for all nodes then this is not supported by Nifi :

https://docs.cloudera.com/cfm/2.0.4/nifi-toolkit-guide/topics/nifi-wildcard_certificates.html

For more information about nifi certificate recommendations please see:

TLS/SSL certificate requirements and recommendations | CDP Private Cloud (cloudera.com)

IF you find this is helpful please accept solution.

Thanks

[#3]

@helk Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @SAMSAL @MattWho  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#4 · Accepted solution]

Hi,

Usually when you get this error it means the certificate is not setup correctly to work with Nifi. For example if you are using wildcard certificate for all nodes then this is not supported by Nifi :

https://docs.cloudera.com/cfm/2.0.4/nifi-toolkit-guide/topics/nifi-wildcard_certificates.html

For more information about nifi certificate recommendations please see:

TLS/SSL certificate requirements and recommendations | CDP Private Cloud (cloudera.com)

IF you find this is helpful please accept solution.

Thanks

[#5 · Accepted solution]

@helk 

You can use a single certificate to secure all your nodes, but i would not recommend doing so for security reasons.  You risk compromising all your host if any one of them is compromised.  Additionally NiFi nodes act as clients and not just servers.  This means that all your hosts will identify themselves as the same client (based off DN).  So tracking client initiated actions back to a specific node would be more challenging.  And if auditing is needed, made very difficult.

The SAN is meant to be used to differently.  Let's assume you host an endpoint searchengine.com which is back by 100 servers to handle client requests.  When a client tries to access searchengine.com that request may get routed to anyone of those 100 servers.  The certificate issues to each of those 100 servers is unique to each server; however, every single one of them will have the searchengine.com as an additional SAN entry in addition to their unique hostname.  This allows the host verification to still be successful since all 100 are also known as searchengine.com.

Your specific issue based on shared output above is caused by the fact that your single certificate does not have "nifi01" in the list of Subject Alternative Names (SAN).  It appears you only added nifi02 and nifi03 as SAN entries. The current hostname verification specs no longer use DN for hostname verification.  Only the SAN entries are used for that.  So all names(hostnames, common names, IPs) that may be used when connecting to a host must be included in the SAN list.

NiFi cluster keystore requirements:
1. keystore can contain only ONE privateKeyEntry.
2. PrivateKey can not use wildcards in the DN.
3. PrivateKey must contain both clientAuth and serverAuth Extended Key Usage (EKU).
4. Privatekey must contain at least one SAN entry matching the hostname of server on which keystore will be used.

The NiFi truststore must contain the complete trust chain for your cluster node's PrivateKeys.  On truststore is typically copied to and used on all nodes.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt