Member since
06-24-2021
6
Posts
0
Kudos Received
0
Solutions
11-23-2021
07:25 AM
@MattWho You were right, the missing rootCA and intermediate certificates missed in the nifi nodes truststore were the cause of the problem(s)! As soon as I added them in each nifi node truststore, it solved my problem and the node were able to communicate and transmit heartbeat through port 11443! Thanks a lot for your time and your help! Best regards Emmanuel
... View more
11-23-2021
01:03 AM
@MattWho Thanks for your response. 1) Each nifi node's truststore does not conaints a separate TrustedCertEntry for each of the three CA certificates of the trust chain. The TrustedCertEntry of node2 and node3 in node1'truststore contain the full chain certification. Also, these three CA certificates are also present and trusted certificates in the JVM truststore (/etc/pki/java/cacert). 2) The acceptable client certificates do need to include CA certificates as well? Even if node certificates already contain rootCA in their full chain? 3) I'll try to convert PKCS12 to JKS, but Nifi support PKCS12 format no? Thanks Emmanuel
... View more
11-17-2021
02:58 AM
@MattWho Here is the keystore.jks output from nifi-node1 : Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
Alias name: nifi-node1
Creation date: Nov 17, 2021
Entry type: PrivateKeyEntry
Certificate chain length: 4
Certificate[1]:
Owner: CN=nifi-node1, ..... C=FR
Issuer: CN=Corporate Service CA 2, ... C=FR
Serial number: xxx
Valid from: Tue Oct 19 13:04:45 CEST 2021 until: Mon Jan 22 12:05:15 CET 2024
Certificate fingerprints:
SHA1: XXXXXX
SHA256: XXXXXX
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 4096-bit RSA key
Version: 3
Extensions:
#1: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
.....
#2: ObjectId: 2.5.29.35 Criticality=false
...
#3: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
....
]
#4: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
[CertificatePolicyId: [xxxxxx]
[] ]
]
#5: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
clientAuth
serverAuth
]
#6: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Key_Encipherment
]
#7: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: nifi-node1
DNSName: nifi-node2
DNSName: nifi-node3
DNSName: load-balancer-fqdn (modified)
DNSName: load-balancer-hostname (modified)
]
#8: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: xxxxx
0010: xxxxxx
]
]
Certificate[2]:
Owner: CN=Corporate Service CA 2, ...., C=FR
Issuer: CN=Corporate CA 1, ...., C=FR
Serial number: xxxxx
Valid from: Fri May 23 02:00:00 CEST 2014 until: Sat May 23 02:00:00 CEST 2026
Certificate fingerprints:
SHA1: XXXXX
SHA256: XXXXX
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 4096-bit RSA key
Version: 3
Extensions:
#1: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
AuthorityInfoAccess [
[
XXXX]
]
#2: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
XXXX
]
]
#3: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:true
PathLen:0
]
#4: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
XXX
]
#5: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
[CertificatePolicyId: [XXXXXXX]
[] ]
[CertificatePolicyId: [XXXXXXX]
[] ]
[CertificatePolicyId: [XXXXXXX]
[] ]
[CertificatePolicyId: [XXXXXXX]
[] ]
]
#6: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
Key_CertSign
Crl_Sign
]
#7: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
XXXXx
]
]
Certificate[3]:
Owner: CN=Corporate CA 1, ..., C=FR
Issuer: CN=Root CA 1, ...., C=FR
Serial number: XXXX
Valid from: Thu Jun 09 02:00:00 CEST 2011 until: Mon Jun 09 02:00:00 CEST 2036
Certificate fingerprints:
SHA1: XXXX
SHA256: XXXX
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 4096-bit RSA key
Version: 3
Extensions:
#1: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
AuthorityInfoAccess [
[
XXX]
]
#2: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
XXXX
]
]
#3: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:true
PathLen:2147483647
]
#4: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
XXXXX
]
#5: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
[CertificatePolicyId: [XXXX]
[] ]
]
#6: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
Key_CertSign
Crl_Sign
]
#7: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
XXXX
]
]
Certificate[4]:
Owner: CN=Root CA 1, ..., C=FR
Issuer: CN=Root CA 1, .... C=FR
Serial number: XXXXX
Valid from: Thu Jun 09 02:00:00 CEST 2011 until: Mon Jun 09 02:00:00 CEST 2036
Certificate fingerprints:
SHA1: XXXXXX
SHA256: XXXX
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 4096-bit RSA key
Version: 3
Extensions:
#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
XXXX
]
]
#2: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:true
PathLen:2147483647
]
#3: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
Key_CertSign
Crl_Sign
]
#4: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
XXX
]
] Nifi version: 1.13.2 Java version: openjdk version "1.8.0_302" OpenJDK Runtime Environment (build 1.8.0_302-b08) OpenJDK 64-Bit Server VM (build 25.302-b08, mixed mode) Regarding certificates/keystore requirements from https://docs.cloudera.com/cfm/2.1.2/cfm-security/topics/cfm-security-tls-certificate-requirements-re... all are verified. Thanks
... View more
11-16-2021
05:00 AM
Hi Thanks for your reply. Each cluster node's truststore contains pem cerificate from other cluster nodes. For example, truststore from node1 contains pem certs from node2 and node3, etc. Is it the question you're asking ?
... View more
11-16-2021
01:18 AM
Hi list, I’m facing a weird problem I can’t resolve or even understand with my secured nifi cluster. Below is the situation. We have setup a secured nifi cluster with 3 nodes, say node1,node2 and node3. For each of theses nodes, we’ve manually created a SSL certificate signing request (CSR) (using a password protected private key) to be signed by our internal CA. Once we’ve get the certificates signed, I’ve installed each node certificates following this procedure: Add the full certificate chain (root + intermediate certificates) into the signed certificate. cat nifi-nodeX.pem cert_chain.pem > full-nifi-nodeX.pem Create a PKCS12 certificate using private key (.key) and full signed certificate (.pem) openssl pkcs12 -export -in full-nifi-nodeX.pem -inkey nifi-nodeX.key -out nifi-nodeX.p12 \ -name nifi-nodeX -passin pass:"XXXXXX" -passout pass:YYYYY; Import nifi-nodeX.p12 into the nifi-nodeX keystore keytool –omportkeystore –deststorepass xxxxxx –destkeystore keystore.jks –srckeystore nifi-nodeX.p12 –srcstoretype PKCS12 Then added each other nifi-node certificates (.pem) into nifi-truststore node1: add full-nifi-node2 + full-nifi-node3 into truststore node2: add full-nifi-node1 + full-nifi-node3 into truststore node3: add full-nifi-node2 + full-nifi-node1 into truststore Restarted each node Once each node are restarted, I can connect to the web UI, but I’ve got an error message saying: For info, web UI is reachable on port 8443 Invalid State: The Flow Controller is initializing the Data Flow. Looking at node logs (nifi-app.log) I can see that each node cannot talk to each other and to the Coordinator to send heartbeat messages: Nifi-node1: INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi-node2:11443; will use this address for sending heartbeat messages INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at nifi-node2:11443. Will send Cluster Connection Request to this address WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed unmarshalling 'CONNECTION_RESPONSE' protocol message from nifi-node2.rd1.rf1/10.108.70.39:11443 due to: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate … Nifi-node2: WARN [Process Cluster Protocol Request-1] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from nifi-node3 due to Empty client certificate chain INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi-node2:11443; will use this address for sending heartbeat messages INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at nifi-node2:11443. Will send Cluster Connection Request to this address WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: java.net.SocketException: Connection reset by peer (Write failed) … Nifi-node3: INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi-node2:11443; will use this address for sending heartbeat messages INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at nifi-node2:11443. Will send Cluster Connection Request to this address WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed unmarshalling 'CONNECTION_RESPONSE' protocol message from nifi-node2/10.108.70.39:11443 due to: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate It looks like the signed certificates are not ok regarding the logs errors. However, trying these certificates using openssl s_client command works as expected: openssl s_client -connect nifi-node3:11443 -cert full-nifi-node3.pem -key nifi-node3.key -pass pass:'XXXXXXX’ CONNECTED(00000003) depth=3 C = FR, O = SAFRAN, OU = 0002 562082909, CN = SAFRAN Root CA 1 verify return:1 depth=2 C = FR, O = SAFRAN, OU = 0002 562082909, CN = SAFRAN Corporate CA 1 verify return:1 depth=1 C = FR, O = SAFRAN, OU = 0002 562082909, CN = SAFRAN Corporate Service CA 2 verify return:1 depth=0 C = FR, O = SAFRAN, OU = SAFRAN SA, OU = 0002 562082909, CN = nifi-node3 verify return:1 --- Certificate chain 0 s:/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node3 i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate Service CA 2 1 s:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate Service CA 2 i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate CA 1 2 s:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate CA 1 i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Root CA 1 3 s:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Root CA 1 i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Root CA 1 --- Server certificate -----BEGIN CERTIFICATE----- …. -----END CERTIFICATE----- subject=/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node3 issuer=/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate Service CA 2 --- Acceptable client certificate CA names /C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node3 /C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=niif-node2 /C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node1 /C=FR/OU=SAFRAN SA/OU=0002 562082909/O=SAFRAN/CN=Safran Nifi Admin /C=FR/OU=SAFRAN SA/OU=0002 562082909/O=SAFRAN/CN=localhost Client Certificate Types: ECDSA sign, RSA sign, DSA sign Requested Signature Algorithms: 0x07+0x08:0x08+0x08:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:0x04+0x08:0x05+0x08:0x06+0x08:0x09+0x08:0x0A+0x08:0x0B+0x08:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1 Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1 Peer signing digest: SHA512 Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 8911 bytes and written 8534 bytes New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 4096 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: 68686A816F510BED151FEBB80604862B799CD0D5DFCEA9602A9E204E9EC5741E Session-ID-ctx: Master-Key: CB4E24EDCAA3518494C04762965452CDC9CE993FCCAF3DBCCF76755376B808667342AF327DE5B8DE6B3B981F55B3CB90 Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1635340626 Timeout : 300 (sec) Verify return code: 0 (ok) --- closed There is something I don’t get! I’ve tried all the above procedure without adding the full cert chain, same errors. I’ve tried with autogenerated self-signed certificates using nifi-toolkit, and it works as expected, so I think there is definitely something wrong with the signed certificates but I’ve no clue at all what it could be. Please could someone light my lantern, I’ve no more idea or way to explore. Regards Emmanuel
... View more
Labels:
- Labels:
-
Apache NiFi