NiFi Error Despite Successful SSL Handshake using openssl s_client

[#1]

I just recently imported a certificate chain into the keystore that NiFi points to, on 3 NiFi nodes, call them node1, node2 and node3. The truststore.jks file so far has been unedited.

Testing out the SSL handshakes between nodes, I get:

SSL handshake has read 4537 bytes and written 495 bytes

...

return code: 0 (ok)

Executed from node 2 requesting node 1 (using the same port configured in NiFi SSL settings in Ambari)

Similarly other combinations also were successful, (node1 -> node2, node3 -> node1, etc.)

However, when after the certificate import and then restarting NiFi, trying the NiFi UI, it shows that the cluster has been disconnected.

Furthermore, it shows that the SSL handshakes are failing:

Attempt to contact NiFi Node https://node2:port/nifi did not complete due to exception: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Attempt to contact NiFi Node https://node3:port/nifi did not complete due to exception: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Attempt to contact NiFi Node https://node1:port/nifi did not complete due to exception: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.

What is going on here? Why isn't the SSL handshake working through NiFi?

[#2]

@mkobe 

You comment that you have not modified your truststore.  What currently exists in the truststore.jks that you are using on all three of your nodes?

keytool -v -list -keystore truststore.jks

NiFi will not use the keystore for verifying certifcate trust.  It will use the configured truststore.

Also sharing the verbose output from one or more of your nodes may help get to the cottom of your issue as well.

keytool -v -list -keystore keystore.jks

Also make sure the following criteria against your keystore.jks:
1. Keystore.jks must contain only ONE PrivateKeyEntry.
2. PrivateKeyEntry does not use wildcards
3. PrivateKeyEntry supports clientAuth and serverAuth Extended Key Usage (EKU)
4. PrivateKeyEntry includes at least one Subject Alternative Name (SAN) entry that matches the hostname of the NiFi node on which the keystore is being used.

Hope this helps,

Matt

[#3]

@MattWho 

Hi Matt,

For your third point : 3. PrivateKeyEntry supports clientAuth and serverAuth Extended Key Usage (EKU) -> do you mean needs both ?

Can you give a situation where we need both attributes ?

I have a nifi cluster with 3 servers for each and in each cert I only have one EKU (serverAuth) but my 3 servers can still communicate between them. Even nifi and nifi-registry can communicate too with only one attribute in EKU field. 

[#4]

@JC_ROS 

With a secured NiFi all actions are authenticated and authorized.  Thus mutual TLS authentication is required for NiFi things like:

1. Node to Node communications (Older versions of NiFi had ability to set NeedClientAuth=false in the nifi.properties to make node to node comms 1-way TLS.  Newer NiFi releases removed this property resulting in mutual TLS being required here.
2. NiFi Remote Process Groups (RPG) - The S2S protocol used by the RPG required mutual TLS.  The NiFI instance running the RPG is acting as a client and must present a clientAuth certificate.
3. NiFi load balanced connections

The above all utilize the NiFi node keystore and truststore. 

Hope this helps,

Matt

[#5]

@JC_ROS 

I struggle with how your NiFi communicates with a NiFi-Registry without clientAuth in your NiFi node's PrivateKeyEntry unless your NiFi-registry has not been secured.

The NiFi nodes must be authorized in your NiFi-Registry to both Read buckets and act as a proxy.  Authorizing a NiFi node requires that the node authenticate itself in the TLS handshake.  Unless of course you have authorized "anonymous".

Thanks,

Matt

[#6]

@MattWho 

Thanks for the answers !

You're right in my nifi.properties : nifi.security.needClientAuth is set to false. My version of nifi is an old one (1.9.2). I will pay more attention during my next upgrade.

My nifi and my nifi-registry are both secured. I have only added the CN of my nifi app cert into authorizers.xml file (userGroupProvider and accessPolicyProvider).