Secure NiFi Site to Site (S2S) on standalone NiFi instance TLS/SSL configuration

[#1]

Hi Guys,

I’m configuring a secure single NiFi instance (v1.3.0) to use Site to Site (S2S) to ingest reporting tasks back to itself (monitor disk usage, memory etc.) but have hit a hurdle. S2S works successfully in unsecure mode but not when I apply security to it.

My environment

• NiFi 1.3.0 secured (LDAP authentication provider) and issue free
• NiFi S2S working perfectly in unsecured mode and ingesting reporting task messages as flow files
• My secure S2S configuration uses a SiteToSiteBullitenReportingTask in RAW mode with a StandardSSLServiceContext (same issue using HTTPS transport protocol)
• The StandardSSLServiceContext is using the exact same keystore and truststore that were used in successfully securing the NiFi UI
• Java details: java version "1.8.0_91", Java(TM) SE Runtime Environment (build 1.8.0_91-tdc1-b14), Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)

In the StandardSSLServiceContext you have the ability to specify the TLS version to use when making the connection. I’ve tried all of the settings but toggle between a broken pipe SocketException or a handshake_failure depending on the choice. I’ve lowered the log level to DEBUG and can see the attempted handshake but my connection is always terminated by the server. Interestingly, even with a working secure NiFi instance I’ve never been able to use the openssl –sclient tool on the same server to successfully test the connection but have never had a problem when accessing NiFi via the web UI. Would the intelligent Hortonworks community have any suggestions that might be able to assist my troubleshooting this issue?

s2s-stacktrace.txt

[#2]

Is there an error that is shown on the Remote Process Group? If so can you provide that error and possibly the stacktrace that goes with it from nifi-app.log?

[#3]

Thanks for the quick reply Bryan. The errors are thrown by the SiteToSiteBulletinReportingTask Reporting Task when trying to use its configured SSL Context Service (StandardSSLContextService) Controller Service to define an input monitor port. The configuration is the same as Pierre Villard's monitoring disk space example here (https://pierrevillard.com/2017/05/13/monitoring-nifi-site2site-reporting-tasks/) except I am using a single NiFi instance to both report and receive the monitor. (Stack trace attached)

[#4 · Accepted solution]

What issue do you get when using openssl s_client ? You should definitely be able to connect using s_client, although you may need to provide -CAfile option to trust the issued server certificate and -cert and -key options to provide a client certificate as below:

openssl s_client -connect nifi.nifi.apache.org:9443 -debug -state -CAfile conf/nifi-cert.pem -cert conf/client.pem -key conf/client.key

As for the handshake_failure issue, this may be because:

1. The certificate has expired or is invalid
2. The provided keystore or key password is incorrect and the controller service cannot access the keystore
3. The server and client cannot agree on a suitable cipher suite during negotiation

Diagnosing with s_client is definitely the correct tool. You can also enable TLS debugging via the bootstrap.conf file with the line:

java.arg.15=-Djavax.net.debug=ssl,handshake

This output will be in logs/nifi-bootstrap.log.

[#5]

Thanks for the logging tip Andy this has been very useful. I had previously tried the openssl s_client command you mention above with the same, cert, key, and CA and connected without issue to another service on the server (knox) so I was confident of valid credentials. The bootstrap log is now showing a SSL_NULL_WITH_NULL_NULL and NiFi Web Server-228, fatal error: 40: no cipher suites in common which I was unable to see before. I'll explore further tomorrow and get back with a confirmation. Thanks again I appreciate the assistance.

[#6]

Hi Andy, unfortunately I'm still having TLS cipher connection compatibility problems. NiFi always appears to present a ClientHello TLSv1.2 message irrespective of my selection on the StandardSSLContextService (v1.3.0) SSL Protocol. I assume the ClientHello is launched by NiFi using a java SSL connection and has no dependency on my underlying openssl version (OpenSSL 0.9.8j-fips)?

[#7]

Hi DH.

As of Apache NiFi 1.2.0, only TLSv1.2 is supported for incoming connections [Release Notes - Security]. This is independent of the version of OpenSSL installed, and unrelated to the selection of TLS/SSL protocol in StandardSSLContextService (see discussion on PR 1986 about removing invalid protocol version options from the dropdown because they are not supported). However, because your client should be the same instance as your server (the SiteToSite reporting task is pointing back to itself, correct?) the same protocol versions and cipher suites should be available to both roles. Can you do a remote debugging session and trace the outgoing connection to determine what protocol versions and cipher suites are available to the SSLContext object at that time?

[#8]

Thanks for the info Andy this is good to know. I'm very close to getting there and now have a successful handshake taking place between NiFi and itself thanks to your useful advice (I've learnt more about SSL debugging in the process than I'd care to admit!). The problem has been the strict format required for the NiFi certs which were issued to us by our CA team and not generated using the NiFi toolkit. This has been complicated by not having a recent version of openssl capable of debugging the TLS1.2 version the Jetty server requires. For interest and other users out there make sure your certs have an EKU of both clientAuth and serverAuth specified (if your EKU is critical) and your KU has DigitalSignature specified. I've now hit the same problem as you've assisted with here so I'll try to work through your suggestions (http://apache-nifi.1125220.n5.nabble.com/NiFi-1-1-1-Secure-Cluster-Setup-Issue-HTTPS-Wrong-Hostname-wrong-should-be-lt-my-ip-address-gt-td15361.html). Thanks again.

[#9]

D H,

You bring up a good point that the EKU restriction should probably be made more evident in the docs. It makes sense -- the certificates are used in both server and client identification roles, so the roles must both be present if either is populated, but many people are unfamiliar with the EKU behavior (or unaware that the certificates serve dual purposes). I hope you are able to resolve your issue quickly.