Secured Nifi Cluster Setup.

[#1]

Hi All,

I am trying to configure the 3node secured Nifi cluster setup by followinng the below Link .

But between nodes the connection not happened after enabled SSL/LDAP and i am getting the below error.

2017-04-01 09:05:47,494 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message due to: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2017-04-01 09:05:47,494 ERROR [Process Cluster Protocol Request-7] o.a.nifi.security.util.CertificateUtils The incoming request did not contain client certificates and thus the DN cannot be extracted. Check that the other endpoint is providing a complete client certificate chain
2017-04-01 09:05:47,494 WARN [Process Cluster Protocol Request-7] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from HKLPATHAS02.example.com due to org.apache.nifi.cluster.protocol.ProtocolException: java.security.cert.CertificateException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
org.apache.nifi.cluster.protocol.ProtocolException: java.security.cert.CertificateException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
        at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.getRequestorDN(SocketProtocolListener.java:221) ~[nifi-framework-cluster-protocol-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
        at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:133) ~[nifi-framework-cluster-protocol-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
        at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136) [nifi-socket-utils-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_102]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
Caused by: java.security.cert.CertificateException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
        at org.apache.nifi.security.util.CertificateUtils.extractPeerDNFromClientSSLSocket(CertificateUtils.java:306) ~[nifi-security-utils-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
        at org.apache.nifi.security.util.CertificateUtils.extractPeerDNFromSSLSocket(CertificateUtils.java:261) ~[nifi-security-utils-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
        at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.getRequestorDN(SocketProtocolListener.java:219) ~[nifi-framework-cluster-protocol-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
        ... 5 common frames omitted
Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
        at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:431) ~[na:1.8.0_102]
        at org.apache.nifi.security.util.CertificateUtils.extractPeerDNFromClientSSLSocket(CertificateUtils.java:291) ~[nifi-security-utils-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
        ... 7 common frames omitted
^C

[#2]

Hi @Anishkumar Valsalam

It's a SSLHandshake Error. Verify the certificates.

Root and Intermediate certificate goes to Truststore. Follow this link

https://community.hortonworks.com/articles/58009/hdf-20-enable-ssl-for-apache-nifi-from-ambari.html

[#3]

@Ram Baskaran Thanks but i confiugred SSL certificate but still it showing error.

I am using selfsigned certificate , is there any way to setup selfsigned?

Note: i have imported the selfsigned cert to keystore also.

[root@HKLPATHAS02 ~]# openssl s_client -connect HKLPATHAS02.example.com:9090
CONNECTED(00000003)
140669500790688:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:184:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 247 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

[#4]

please find the openssl connection output, now it looks okay but still i am getting the above handshake error.

[root@HKLPATHAS03 bin]# openssl s_client -connect HKLPATHAS03.example.com:9090
CONNECTED(00000003)
depth=0 C = HK, ST = HK, L = HK, O = EBB, OU = SAAS, CN = HKLPATHAS03.example.com
verify error:num=18:self signed certificate
verify return:1
depth=0 C = HK, ST = HK, L = HK, O = EBB, OU = SAAS, CN = HKLPATHAS03.example.com
verify return:1
---
Certificate chain
 0 s:/C=HK/ST=HK/L=HK/O=EBB/OU=SAAS/CN=HKLPATHAS03.example.com
   i:/C=HK/ST=HK/L=HK/O=EBB/OU=SAAS/CN=HKLPATHAS03.example.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDQzCCAwGgAwIBAgIERreCwDALBgcqhkjOOAQDBQAwczELMAkGA1UEBhMCSEsx
CzAJBgNVBAgTAkhLMQswCQYDVQQHEwJISzEMMAoGA1UEChMDU0NCMQ0wCwYDVQQL
EwRIQUFTMS0wKwYDVQQDEyRIS0xQQVRIQVMwMy5oay5zdGFuZGFyZGNoYXJ0ZXJl
ZC5jb20wHhcNMTcwNDAxMTY1MzAwWhcNMTkwNDAxMTY1MzAwWjBzMQswCQYDVQQG
EwJISzELMAkGA1UECBMCSEsxCzAJBgNVBAcTAkhLMQwwCgYDVQQKEwNTQ0IxDTAL
BgNVBAsTBEhBQVMxLTArBgNVBAMTJEhLTFBBVEhBUzAzLmhrLnN0YW5kYXJkY2hh
cnRlcmVkLmNvbTCCAbgwggEsBgcqhkjOOAQBMIIBHwKBgQD9f1OBHXUSKVLfSpwu
7OTn9hG3UjzvRADDHj+AtlEmaUVdQCJR+1k9jVj6v8X1ujD2y5tVbNeBO4AdNG/y
ZmC3a5lQpaSfn+gEexAiwk+7qdf+t8Yb+DtX58aophUPBPuD9tPFHsMCNVQTWhaR
MvZ1864rYdcq7/IiAxmd0UgBxwIVAJdgUI8VIwvMspK5gqLrhAvwWBz1AoGBAPfh
oIXWmz3ey7yrXDa4V7l5lK+7+jrqgvlXTAs9B4JnUVlXjrrUWU/mcQcQgYC0SRZx
I+hMKBYTt88JMozIpuE8FnqLVHyNKOCjrh4rs6Z1kW6jfwv6ITVi8ftiegEkO8yk
8b6oUZCJqIPf4VrlnwaSi2ZegHtVJWQBTDv+z0kqA4GFAAKBgQD69XAoviqJdSxO
74nFUTw5+MDMucIYztGW+DcmkCQPta9K7jQpJ0TLTIrrYrt/81OLoZM34PeYOy94
aBZnW7cjaAG6D+SSeUaz4n5ZeD5LGLK3BbvWcPO/glxo8wN0nBwnZl8fW1/9docp
TV3mzWb51lA9L71hMsyQDArGyy1Sl6MhMB8wHQYDVR0OBBYEFO9pdvOKeeVW8EOm
avaPlFnBNaP0MAsGByqGSM44BAMFAAMvADAsAhQ4hVHePJIWjPCsVBLuBxS5Zjxx
fgIUFyFOfoModKJ+rH5Io4BQGYNRdpA=
-----END CERTIFICATE-----
subject=/C=HK/ST=HK/L=HK/O=EBB/OU=SAAS/CN=HKLPATHAS03.example.com
issuer=/C=HK/ST=HK/L=HK/O=EBB/OU=SAAS/CN=HKLPATHAS03.example.com
---
Acceptable client certificate CA names
/C=HK/ST=HK/L=HK/O=EBB/OU=SAAS/CN=HKLPATHAS01.example.com/emailAddress=EDM-Hadoop-Admin@example.com
/C=HK/ST=HK/L=HK/O=EBB/OU=SAAS/CN=HKLPATHAS03.example.com
/C=HK/ST=HK/L=HK/O=EBB/OU=SAAS/CN=HKLPATHAS03.example.com/emailAddress=EDM-Hadoop-Admin@example.com
/C=HK/ST=HK/L=HK/O=EBB/OU=SAAS/CN=HKLPATHAS02.example.com/emailAddress=EDM-Hadoop-Admin@example.com
Server Temp Key: DH, 1024 bits
---
SSL handshake has read 1984 bytes and written 489 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-DSS-AES256-SHA256
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : DHE-DSS-AES256-SHA256
    Session-ID: 58DFF03F0545CC5EC538987F39998949FA771A3C622A2C3FD8954ECC61375B5B
    Session-ID-ctx:
    Master-Key: 78E19935E586096223FEA58984286A08CFD7F8DD34D1457563B4CA70A1402254A6A3114AF6D73A4510932F9629F28C15
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1491071039
    Timeout   : 300 (sec)
    Verify return code: 18 (self signed certificate)
---

[#5]

@Matt Clarke @Bryan Bende @Pierre Villard Can you help on this issue.

[#6]

You need to tell s_client that a self-signed certificate is ok by providing the -CAfile flag and the path to the CA's certificate, exported as PEM.

$ openssl s_client -connect HKLPATHAS03.example.com:9090 -CAfile /opt/certs/ca.pem

[#7 · Accepted solution]

The error is saying that one node in the cluster is attempting to make a heartbeat connection to the other node, but it is not providing a valid client certificate in order to authenticate itself during the TLS handshake negotiation. There are a few possible reasons for this error:

• The node is not sending the client certificate. Ensure that nifi.security.needClientAuth=true and nifi.cluster.protocol.is.secure=true are present in your nifi.properties file.
• The truststore on the receiving node does not contain the public key certificate of the connecting node. When you followed the instructions from that link, how did you generate the respective certificates? Using the Apache NiFi TLS Toolkit as described by Pierre should ensure that all node certificates are signed by the same CA and that the CA is imported into the common truststore. If you manually generated your certificates, ensure that they are trusted on each node (you can do this with OpenSSL's s_client tool).

[#8]

@Andy LoPresto

Thanks much Tls toolkit Resolved my issue :-)

[#9]

We are trying to set up a 3 node nifi cluster on GCP virtual machine (Ubuntu). Have used a CA signed certificate for creating truststore and keystore (Followed this link for creation). Have attached them through nifi.properties file. Still getting below exception on tailing logs:

2022-05-26 18:14:26,544 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at nifi-hostname:7474. Will send Cluster Connection Request to this address
2022-05-26 18:14:26,780 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-05-26 18:14:31,783 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi-hostname:7474; will use this address for sending heartbeat messages

On the nifi UI seeing this 

javax.net.ssl.SSLPeerUnverifiedException: Hostname xx.yy.aa.bb not verified: certificate: sha256/I6cvWHqdHyhMxgNMGFcIwjY2zssGR***hidding_something_here**wjnWezSDm4= DN: CN=Guru Prakash, OU=Comm***, O=**pna, L=Bangalore, ST=Karnataka, C=IN subjectAltNames: []

Please help 🙏