NIFI S2S communication issue between two secured NIFI instances

[#1]

Hi

I am trying to setup S2S communication protocol between two secured NIFI instances.

I have secured each NIFI instance with SSL Authentication.

Establishing connection is failing on RPG with error "due to java.io.IOException: Tried all cluster URLs but none of those was accessible"

Followed the steps (Option 2) mentioned in https://community.hortonworks.com/articles/88473/site-to-site-communication-between-secured-https-a....

Attached nifi-app.txt rpgerror.png.

Could somebody please point me in the right direction?

Thank you

[#2]

@Matt Clarke

I am getting below error on Remote Process Group when i try to setup Site to Site between 2 secured NIFI instances.

keystore.jks and truststore.jks files are available in each node of the cluster. But still error is saying that unable to find valid certification path to requested target.

Could you please point me in the right direction?

2018-05-21 10:00:15,229 WARN [Remote Process Group 7484c56b-0163-1000-ffff-ffff89fed68c: https://hdfnifidev02.medassurant.local:9091/nifi/,https://hdfnifidev03.medassurant.local:9091/nifi/,... Thread-1] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from https://hdfnifidev02.medassurant.local:9091/nifi-api due to javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2018-05-21 10:00:15,257 WARN [Remote Process Group 7484c56b-0163-1000-ffff-ffff89fed68c: https://hdfnifidev02.medassurant.local:9091/nifi/,https://hdfnifidev03.medassurant.local:9091/nifi/,... Thread-1] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from https://hdfnifidev03.medassurant.local:9091/nifi-api due to javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2018-05-21 10:00:15,287 WARN [Remote Process Group 7484c56b-0163-1000-ffff-ffff89fed68c: https://hdfnifidev02.medassurant.local:9091/nifi/,https://hdfnifidev03.medassurant.local:9091/nifi/,... Thread-1] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from https://hdfnifidev04.medassurant.local:9091/nifi-api due to javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2018-05-21 10:00:15,287 WARN [Remote Process Group 7484c56b-0163-1000-ffff-ffff89fed68c: https://hdfnifidev02.medassurant.local:9091/nifi/,https://hdfnifidev03.medassurant.local:9091/nifi/,... Thread-1] o.a.n.remote.StandardRemoteProcessGroup Unable to connect to RemoteProcessGroup[https://hdfnifidev02.medassurant.local:9091/nifi/,https://hdfnifidev03.medassurant.local:9091/nifi/,https://hdfnifidev04.medassurant.local:9091/nifi/] due to java.io.IOException: Tried all cluster URLs but none of those was accessible. Last Exception was javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2018-05-21 10:00:19,862 WARN [Timer-Driven Process Thread-4] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from https://hdfnifidev02.medassurant.local:9091/nifi-api due to javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2018-05-21 10:00:19,889 WARN [Timer-Driven Process Thread-4] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from https://hdfnifidev03.medassurant.local:9091/nifi-api due to javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2018-05-21 10:00:19,918 WARN [Timer-Driven Process Thread-4] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from https://hdfnifidev04.medassurant.local:9091/nifi-api due to javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2018-05-21 10:00:19,919 WARN [Timer-Driven Process Thread-4] o.apache.nifi.controller.FlowController Unable to communicate with remote instance RemoteProcessGroup[https://hdfnifidev02.medassurant.local:9091/nifi/,https://hdfnifidev03.medassurant.local:9091/nifi/,https://hdfnifidev04.medassurant.local:9091/nifi/] due to org.apache.nifi.controller.exception.CommunicationsException: Unable to communicate with Remote NiFi at URI https://hdfnifidev02.medassurant.local:9091/nifi/,https://hdfnifidev03.medassurant.local:9091/nifi/,... due to: Tried all cluster URLs but none of those was accessible. Last Exception was javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2018-05-21 10:00:20,053 INFO [Clustering Tasks Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-05-21 10:00:19,912 and sent to rsdevhdf2.medassurant.local:9088 at 2018-05-21 10:00:20,053; send took 141 millis

[#3 · Accepted solution]

@Srini K

Can you share the verbose output from your keystore and truststore on each side of this connection?

keytool -v -list -keystore <your keystore/truststore jks file>

Your keystore should contain only one "PrivateKeyEntry"

--- Check that your PrivateKeyEntry has an extended key usage that supports both "ClientAuth" and "ServerAuth"

--- Verify that whomever is the "issuer" of your PrivateKeyEntry exists as a trustedCertEntry in the truststore on the other NiFi instance.

In above example you can see owner DN is "CN=nifi-sme-14.openstacklocal, OU=NIFI" and it was issued/signed by "CN=nifi-sme-26.openstacklocal, OU=NIFI"

-

Your truststore would contain 1 to many "trustedCertEntry"

--- This file must contain at a minimum a TrustedCertEntry for the issuer of any privateKeyEntry being used to communicate with this system.

- You can see here that my truststore contains a trustedCertEntry for the issuer of my cert on the other NiFi.

-

Thank you,

Matt

-

If you found this answer addressed your question, please take moment to login and click "accept" below the answer.

[#4]

@Matt Clarke Thank you so much for your response. I have attached verbose output of keystore and truststore for both the nifi instances.

Thank You.

[#5]

@Srini K

*** Forum tip: try to avoid responding to an existing answer by starting a new answer. Response may end up being in no specific logical order and become hard to follow. Instead respond to an existing answer by clicking "Add comment" on an existing answer.

-

Looking at your provide keystore and truststore verbose output, TLS/SSL 2-way authentication is going to fail here. Each of your NiFi server's certificate were signed by different Certificate Authorities (CA). So NiFi-2 cannot trust the client cert being presented from NiFi-1 and NiFi-1 cannot trust the cert being presented from NiFi-2.

-

You will need to create a new truststore that contain both "trustedCertEntries" (So basically merging both your truststores together). Then configure both NiFi instance to use that new truststore.

-

Thank you,

Matt

[#6]

Thank you @Matt Clarke

So I just need to merge trustedCertEntries in truststore and keep everything else as it is like below.

nifi1:

Entry type: trustedCertEntry

Owner: CN=hdfnifidev04.medassurant.local, OU=NIFI

Issuer: CN=rsdevhdf3.medassurant.local, OU=NIFI

nifi2:

Entry type: trustedCertEntry

Owner: CN=rsdevhdf3.medassurant.local, OU=NIFI

Issuer: CN=hdfnifidev04.medassurant.local, OU=NIFI

Then I need to copy this truststore file in nifi-toolkit/bin folder on both the nifi clusters.

Then I need to restart the nifi services on both the clusters with NiFi CA Force Regenerate? as true.

Please let me know if I am missing any step.

Do i need to do anything with nifi-cert.pem files between the two nifi clusters.

Thank You.

[#7]

@Srini K

Based on above comment, it looks like you merged the public key for your private key entries which is not what you shodul have done.

-

The new truststore you are trying to generate should contain 2 entries

Your keystore contains 2 entry

Alias name: nifi-cert1
Creation date: May 17, 2018
Entry type: trustedCertEntry

Owner: CN=hdfnifidev04.medassurant.local, OU=NIFI
Issuer: CN=hdfnifidev04.medassurant.local, OU=NIFI
Serial number: 16350a94aa200000000
Valid from: Fri May 11 15:24:22 EDT 2018 until: Mon May 10 15:24:22 EDT 2021
Certificate fingerprints:
         MD5:  A9:BA:53:66:C3:E0:71:EA:94:F0:E2:70:40:F7:B1:58
         SHA1: 3C:48:FA:D4:13:C9:CB:6C:D0:74:08:89:32:9B:A3:B9:86:87:0E:49
         SHA256: B0:24:27:C3:CA:BB:5F:E7:F7:10:45:62:0E:FC:02:2E:11:08:E8:EA:AA:EB:97:89:B3:63:D9:BF:E7:64:A2:5F
         Signature algorithm name: SHA256withRSA
         Version: 3
.....


Alias name: nifi-cert2
Creation date: May 17, 2018
Entry type: trustedCertEntry

Owner: CN=rsdevhdf3.medassurant.local, OU=NIFI
Issuer: CN=rsdevhdf3.medassurant.local, OU=NIFI
Serial number: 162c0b11dfb00000000
Valid from: Fri Apr 13 16:27:35 EDT 2018 until: Mon Apr 12 16:27:35 EDT 2021
Certificate fingerprints:
         MD5:  0B:E6:9B:74:F3:44:97:93:C6:B6:F7:C2:8D:C4:43:14
         SHA1: E0:3F:B4:96:9A:4C:2B:94:AF:85:0D:DA:70:B5:40:B0:AD:B3:9D:4A
         SHA256: D1:B0:3B:CE:F1:2F:DA:C0:53:49:B9:21:A4:A2:A0:B8:7D:75:78:A1:8C:9E:3A:2B:D0:92:7B:05:66:FE:9E:EF
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
......

This new truststore.jks file would need to be copied to both NiFi instance. It would replace the existing truststore.jks being used by those clusters.
-

Guessing you have used Ambari to install these NiFi instances?
**** DO NOT check the box for "NiFi CA Force Regenerate?" <--- Doing so will trigger Ambari to archive the existing keystore.jks and truststore,jks files an run the toolkit to regenerate all new of both.
-

Note: The NiFi CA (nifi TLS toolkit) was never designed to be used in a production environment. It was designed to facilitate an easy way for users to quickly standup a secured NiFi for testing purposes.

-

Yes, a NiFi restart will be needed so that NiFi reads the new truststore.jks file you dropped on each NiFi instance.

-

You do not need to do anything with the keystore.jks files being used by each of your NiFi instances.

-

Thank you,

Matt

[#8]

Thank you @Matt Clarke

I tried but am not able to merge the truststore files using keytool. Can you please explain how to add a new entry in existing truststore file.

Also You mentioned, The NiFi CA is not to be used in production environment. Is there any specific reason? Can you please suggest which option would be recommended to secure NIFI.

Thanks.

[#9]

@Srini K

Managing TLS/SSL issued certificates is not a job for NiFi. The NIFi CA is not backed by any type of long time management capability. Is becomes complicated to deploy and manage across multiple NiFi deployments. You are experiencing some of that management difficulty here now.

-

1. Every Ambari based NiFi installation sets up its own CA. So it becomes difficult to setup communication between systems where each has a different Unique CA signing their certificates. You are trying to get two NiFi's to talk to one another, but this difficulty would extend to any other external service that NiFi needs to communicate to or receive connections from over TLS/SSL.

2. There is no build in management process to handle certificate renews or notifications of expiration. Your NiFi system may be working fine one day in production and then stop working all together the next because your certs expired.

-

In a production managed environment, a corporately or external managed CA should be used to issue, sign, and manage all your certificate needs.

-

Yes, NiFi requires TLS/SSL certificates in order to secure NiFi, but SSL/TLS is not a product of NiFi.

-

As far as merging the content of your two truststore in to a new truststore...

A truststore cannot contain multiple keys with the same alias. Each entry must have a unique alias. The trustedCertEntries in each of your existing truststores have the same alias, so you are going to need to change the alias of one of them. Following commands will extract the trustedCertEntry from each source truststore.jks and put tehem in a new-truststore.jks with new unique alias fro each of those entries:

keytool -importkeystore -srckeystore truststore.jks -destkeystore new-truststore.jks -srcalias nifi-cert -destalias nifi-cert1 -srcstorepass ****-deststorepass ****

keytool -importkeystore -srckeystore truststore.jks -destkeystore new-truststore.jks -srcalias nifi-cert -destalias nifi-cert2 -srcstorepass ****-deststorepass ****