Nifi secured cluster can't send heartbeat betweennodes

[#1]

Hi list,

I’m facing a weird problem I can’t resolve or even understand with my secured nifi cluster. Below is the situation.

We have setup a secured nifi cluster with 3 nodes, say node1,node2 and node3.

For each of theses nodes, we’ve manually created a SSL certificate signing request (CSR) (using a password protected private key) to be signed by our internal CA.

Once we’ve get the certificates signed, I’ve installed each node certificates following this procedure:

• Add the full certificate chain (root + intermediate certificates) into the signed certificate.
  cat nifi-nodeX.pem cert_chain.pem  > full-nifi-nodeX.pem
• Create a PKCS12 certificate using private key (.key) and full signed certificate (.pem)

openssl pkcs12 -export  -in full-nifi-nodeX.pem  -inkey nifi-nodeX.key  -out nifi-nodeX.p12 \

                    -name nifi-nodeX -passin pass:"XXXXXX" -passout pass:YYYYY;

• Import nifi-nodeX.p12 into the nifi-nodeX keystore
  keytool –omportkeystore –deststorepass xxxxxx –destkeystore keystore.jks –srckeystore nifi-nodeX.p12 –srcstoretype PKCS12
• Then added each other nifi-node certificates (.pem) into nifi-truststore
  node1: add full-nifi-node2 + full-nifi-node3 into truststore
  node2: add full-nifi-node1 + full-nifi-node3 into truststore
  node3: add full-nifi-node2 + full-nifi-node1 into truststore
• Restarted each node

Once each node are restarted, I can connect to the web UI, but I’ve got an error message saying:

For info, web UI is reachable on port 8443

Invalid State:

The Flow Controller is initializing the Data Flow.

Looking at node logs (nifi-app.log) I can see that each node cannot talk to each other and to the Coordinator to send heartbeat messages:

Nifi-node1:

INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi-node2:11443; will use this address for sending heartbeat messages

INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at nifi-node2:11443. Will send Cluster Connection Request to this address

WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed unmarshalling 'CONNECTION_RESPONSE' protocol message from nifi-node2.rd1.rf1/10.108.70.39:11443 due to: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate

…

Nifi-node2:

WARN [Process Cluster Protocol Request-1] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from nifi-node3 due to Empty client certificate chain

INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi-node2:11443; will use this address for sending heartbeat messages

INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at nifi-node2:11443. Will send Cluster Connection Request to this address

WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: java.net.SocketException: Connection reset by peer (Write failed)

…

Nifi-node3:

INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi-node2:11443; will use this address for sending heartbeat messages

INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at nifi-node2:11443. Will send Cluster Connection Request to this address

WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed unmarshalling 'CONNECTION_RESPONSE' protocol message from nifi-node2/10.108.70.39:11443 due to: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate

It looks like the signed certificates are not ok regarding the logs errors.

However, trying these certificates using openssl s_client command works as expected:

openssl s_client -connect nifi-node3:11443 -cert full-nifi-node3.pem -key nifi-node3.key -pass pass:'XXXXXXX’

CONNECTED(00000003)

depth=3 C = FR, O = SAFRAN, OU = 0002 562082909, CN = SAFRAN Root CA 1

verify return:1

depth=2 C = FR, O = SAFRAN, OU = 0002 562082909, CN = SAFRAN Corporate CA 1

verify return:1

depth=1 C = FR, O = SAFRAN, OU = 0002 562082909, CN = SAFRAN Corporate Service CA 2

verify return:1

depth=0 C = FR, O = SAFRAN, OU = SAFRAN SA, OU = 0002 562082909, CN = nifi-node3

verify return:1

---

Certificate chain

0 s:/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node3

   i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate Service CA 2

1 s:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate Service CA 2

   i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate CA 1

2 s:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate CA 1

   i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Root CA 1

3 s:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Root CA 1

   i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Root CA 1

---

Server certificate

-----BEGIN CERTIFICATE-----

….

-----END CERTIFICATE-----

subject=/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node3

issuer=/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate Service CA 2

---

Acceptable client certificate CA names

/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node3

/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=niif-node2

/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node1

/C=FR/OU=SAFRAN SA/OU=0002 562082909/O=SAFRAN/CN=Safran Nifi Admin

/C=FR/OU=SAFRAN SA/OU=0002 562082909/O=SAFRAN/CN=localhost

Client Certificate Types: ECDSA sign, RSA sign, DSA sign

Requested Signature Algorithms: 0x07+0x08:0x08+0x08:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:0x04+0x08:0x05+0x08:0x06+0x08:0x09+0x08:0x0A+0x08:0x0B+0x08:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1

Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1

Peer signing digest: SHA512

Server Temp Key: ECDH, P-256, 256 bits

---

SSL handshake has read 8911 bytes and written 8534 bytes

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384

Server public key is 4096 bit

Secure Renegotiation IS supported

Compression: NONE

Expansion: NONE

No ALPN negotiated

SSL-Session:

    Protocol  : TLSv1.2

    Cipher    : ECDHE-RSA-AES256-GCM-SHA384

    Session-ID: 68686A816F510BED151FEBB80604862B799CD0D5DFCEA9602A9E204E9EC5741E

    Session-ID-ctx:

    Master-Key: CB4E24EDCAA3518494C04762965452CDC9CE993FCCAF3DBCCF76755376B808667342AF327DE5B8DE6B3B981F55B3CB90

    Key-Arg   : None

    Krb5 Principal: None

    PSK identity: None

    PSK identity hint: None

    Start Time: 1635340626

    Timeout   : 300 (sec)

    Verify return code: 0 (ok)

---

closed

There is something I don’t get!

I’ve tried all the above procedure without adding the full cert chain, same errors.

I’ve tried with autogenerated self-signed certificates using nifi-toolkit, and it works as expected, so I think there is definitely something wrong with the signed certificates but I’ve no clue at all what it could be.

Please could someone light my lantern, I’ve no more idea or way to explore.

Regards

Emmanuel

[#2]

Hello @emmanuel , 

Have you updated the truststore as well on the master node and distributed the same across different nifi nodes?

[#3]

Hi

Thanks for your reply.

Each cluster node's truststore contains pem cerificate from other cluster nodes. For example, truststore from node1 contains pem certs from node2 and node3, etc.

Is it the question you're asking ? 

[#4]

@emmanuel 

Can you share the verbose output for your NiFi keystore:

keytool -v -list -keystore <nifi keystore>

Does that output align with the certificate requirements for NiFi?
https://docs.cloudera.com/cfm/2.1.2/cfm-security/topics/cfm-security-tls-certificate-requirements-re...

What version of NiFi are your running?

What version of Java is your NiFi using?

If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.

Thank you,

Matt

[#5]

@MattWho 

Here is the

keystore.jks

output from

nifi-node1

:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: nifi-node1
Creation date: Nov 17, 2021
Entry type: PrivateKeyEntry
Certificate chain length: 4
Certificate[1]:
Owner: CN=nifi-node1, ..... C=FR
Issuer: CN=Corporate Service CA 2, ... C=FR
Serial number: xxx
Valid from: Tue Oct 19 13:04:45 CEST 2021 until: Mon Jan 22 12:05:15 CET 2024
Certificate fingerprints:
         SHA1: XXXXXX
         SHA256: XXXXXX
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 4096-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
.....
#2: ObjectId: 2.5.29.35 Criticality=false
...
#3: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
....
]

#4: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
  [CertificatePolicyId: [xxxxxx]
[]  ]
]

#5: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  clientAuth
  serverAuth
]

#6: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Key_Encipherment
]

#7: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: nifi-node1
  DNSName: nifi-node2
  DNSName: nifi-node3
  DNSName: load-balancer-fqdn (modified)
  DNSName: load-balancer-hostname (modified)
]

#8: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: xxxxx
0010: xxxxxx
]
]

Certificate[2]:
Owner: CN=Corporate Service CA 2, ...., C=FR
Issuer: CN=Corporate CA 1, ...., C=FR
Serial number: xxxxx
Valid from: Fri May 23 02:00:00 CEST 2014 until: Sat May 23 02:00:00 CEST 2026
Certificate fingerprints:
         SHA1: XXXXX
         SHA256: XXXXX
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 4096-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
AuthorityInfoAccess [
  [
XXXX]
]

#2: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
XXXX
]
]

#3: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:0
]

#4: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
 XXX
]

#5: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
  [CertificatePolicyId: [XXXXXXX]
[]  ]
  [CertificatePolicyId: [XXXXXXX]
[]  ]
  [CertificatePolicyId: [XXXXXXX]
[]  ]
  [CertificatePolicyId: [XXXXXXX]
[]  ]
]

#6: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  Key_CertSign
  Crl_Sign
]

#7: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
XXXXx
]
]

Certificate[3]:
Owner: CN=Corporate CA 1, ..., C=FR
Issuer: CN=Root CA 1, ...., C=FR
Serial number: XXXX
Valid from: Thu Jun 09 02:00:00 CEST 2011 until: Mon Jun 09 02:00:00 CEST 2036
Certificate fingerprints:
         SHA1: XXXX
         SHA256: XXXX
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 4096-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
AuthorityInfoAccess [
  [
XXX]
]

#2: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
XXXX
]
]

#3: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:2147483647
]

#4: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
XXXXX
]

#5: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
  [CertificatePolicyId: [XXXX]
[]  ]
]

#6: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  Key_CertSign
  Crl_Sign
]

#7: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
XXXX
]
]

Certificate[4]:
Owner: CN=Root CA 1, ..., C=FR
Issuer: CN=Root CA 1, .... C=FR
Serial number: XXXXX
Valid from: Thu Jun 09 02:00:00 CEST 2011 until: Mon Jun 09 02:00:00 CEST 2036
Certificate fingerprints:
         SHA1: XXXXXX
         SHA256: XXXX
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 4096-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
XXXX
]
]

#2: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:2147483647
]

#3: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  Key_CertSign
  Crl_Sign
]

#4: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
XXX
]
]

Nifi version: 1.13.2

Java version: 

openjdk version "1.8.0_302"
OpenJDK Runtime Environment (build 1.8.0_302-b08)
OpenJDK 64-Bit Server VM (build 25.302-b08, mixed mode)

Regarding certificates/keystore requirements from https://docs.cloudera.com/cfm/2.1.2/cfm-security/topics/cfm-security-tls-certificate-requirements-re... all are verified.

Thanks

[#6 · Accepted solution]

@emmanuel 

Does your truststore on each NiFi host contains a separate TrustedCertEntry for each of your three CA certificates in your trust chain?

Your openssl command output does not list your root or intermediate CAs:

Acceptable client certificate CA names

/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node3

/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=niif-node2

/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node1

/C=FR/OU=SAFRAN SA/OU=0002 562082909/O=SAFRAN/CN=Safran Nifi Admin

/C=FR/OU=SAFRAN SA/OU=0002 562082909/O=SAFRAN/CN=localhost

Have you tried converting your PKCS12 keystore to a JKS keystore?

Matt

[#7]

@MattWho 

Thanks for your response.

1) Each nifi node's truststore does not conaints a separate TrustedCertEntry for each of the three CA certificates of the trust chain.

The TrustedCertEntry of node2 and node3 in node1'truststore contain the full chain certification.

Also, these three CA certificates are also present and trusted certificates in the JVM truststore (/etc/pki/java/cacert).

2) The acceptable client certificates do need to include CA certificates as well? Even if node certificates already contain rootCA in their full chain?

3) I'll try to convert PKCS12 to JKS, but Nifi support PKCS12 format no?

Thanks

Emmanuel

[#8]

@emmanuel 

Yes, Apache NiFi support PKCS12 keystores; howevere, JKS is more commonly used format and was making a suggestion to test using JKS to rule out an issue there.

Something else you may want to do is enable debug logging for TLS in NiFi-Registry.
You can accomplish this by adding an additional line to the bootstrap.conf file:

java.arg.<unique num>=-Djavax.net.debug=ssl,handshake

Maybe you are having a cipher compatibility issue?  Maybe connection is trying to use TLS 1.3 and that needs to be disabled so 1.2 is negotiated?

Java's default cacerts file is not used by the NiFi core during Mutual TLS negotiation.  Only the configured truststore configured in the nifi.properties and nifi-registry.properties file is used.

Hope this helps in your investigation,

Matt

[#9]

@MattWho 

You were right, the missing rootCA and intermediate certificates missed in the nifi nodes truststore were the cause of the problem(s)!

As soon as I added them in each nifi node truststore, it solved my problem and the node were able to communicate and transmit heartbeat through port 11443!

Thanks a lot for your time and your help!

Best regards

Emmanuel