Support Questions

Find answers, ask questions, and share your expertise

Truststore Keystore Certificate error when using non-self signed certificate

avatar

I have a server (windows 2012) running nifi. The self-signed cert works fine and I can put a client certificate at each browser to allow access.

I'm trying to utilize a certificate signed by my root authority and issuing agency from my company.

I created a keystore and imported my assigned certificate with private key. I then created a truststore that contains the trusted certs for my root authority and issuer.

When browsing to the site with the updated truststore and keystore jks files, it gives me an error that the client provided a bad cert. The browser has a certificate installed from the issuing agency and root CA and this works with other websites in the domain.

Every example I seem to see around is stating that a self-signed certificate must be used for nifi when using ssl. I can't seem to find a way to utilize a non-self signed certificate.

I've created the keystore and truststore about 100 times in multiple different configurations. Is the problem that I don't have a SAN in the cert? the FQDN is there already. I've spent too many hours trying to fix this and I'm at my wits end here.

8 REPLIES 8

avatar

Hi Justin,

Let's assign some identifiers so we're talking about the same components:

  • Root CA (your company's root certificate authority) -- Root CA
  • Issuing Authority (your company's intermediate CA?) -- Intermediate CA
  • Public certificate identifying the NiFi service signed by Root CA (and/or Intermediate CA) -- nifi.pem
  • NiFi Server
    • Keystore containing externally-signed certificate (nifi.pem) & private key -- nifi.jks
    • Truststore containing externally-signed certificate (nifi.pem) (and/or certificate of Root CA) -- nifi_trust.jks
  • Client
    • Keystore (or PEM files) containing public certificate & private key of client -- client.p12

Steps to identify the issue:

  1. If you can copy the exact error message you get from NiFi's UI when browsing with the client certificate, that will be helpful.
  2. The server truststore (nifi_trust.jks) must contain the exact certificate (or one in the chain that signed it) presented by the client. If your client certificates are still signed by the old, self-signed NiFi certificate, and NiFi's truststore no longer contains that certificate, the client certs will be rejected.
  3. NiFi does not need to use self-signed certificates at all. The TLS Toolkit is provided as a convenience tool for users who do not have a dedicated security/IT team or feel comfortable generating their own CA and certificates manually, and the toolkit does generate self-signed certs. However, any version of NiFi can use externally-signed certificates, and in current Apache NiFi master, the toolkit can even accept externally-signed certificates in place of its own self-signed CA certificate to allow for chained trust.
  4. A SAN must be present in the server certificate for modern browsers to verify the identity of the server (see RFC 6125). NiFi does enforce this when possible.
  5. You can verify that the TLS handshake negotiation works by following these steps:
    1. Convert the client keystore to individual PEM-encoded files (if not already):
      1. Export the public certificate from the PKCS12 keystore: openssl pkcs12 -in client.p12 -out client.der -nodes -password "pass:<client keystore password>"
      2. Export the private certificate from the PKCS12 keystore: openssl pkcs12 -in client.p12 -nodes -nocerts -out client.key -password "pass:<client keystore password>"
      3. If the client.der file contains "-----BEGIN PRIVATE KEY-----", remove the private key block, and rename the file client.pem
      4. If the client.der file does not contain that string, re-encode it as PEM: openssl x509 -inform der -in client.der -out client.pem
    2. Use the openssl s_client command to attempt a connection using the certificates and key: $ openssl s_client -connect <host:port> -debug -state -cert client.pem -key client.key -CAfile <path_to_your_CA_cert.pem>

Please report back with the output of those steps and any additional information that may arise from what I've described. If I misunderstood your scenario or you've already tried this, please let me know.

avatar
Guru

+1 for detailed answer !

avatar

Hi Andy,

Thank you for helping to add some identifiers to that. I'll try to clarify as I go.

My company has a public Root CA and an Intermediate CA. They have authorized me one certificate for my server.

The nifi server currently has an externally signed certificate and private key (to the best of my abilities, that is the case)

Trust store has an externally signed certificate (public Root CA)

The client has only the keys of the RootCA and Intermediate CA.

In Chrome, the first error I got was: ERR_BAD_SSL_CLIENT_AUTH_CERT. I fear this was because I first created the certificate in IIS and tried to move it into a keystore... poorly. (this is new to me)

I had my company generate me another possibility of a request. I used keytool to create the keystore and successfully got the CSR to go through. Then, I got stuck at another error:err_ssl_version_or_cipher_mismatch.

I'm not sure if that is moving forward or backwards at this point in time.

The trust store has the rootCA in it from the company. They apparently have two rootCAs so I added both of them in there. I noticed when I found a handy setting https://community.hortonworks.com/answers/120970/view.html

java.arg.15=-Djavax.net.debug=ssl,handshake 

to get far more detailed information out of the logs. It showed me that even though I had the trust store defined in the properties file, it was still pulling only the trust store from the java directory instead of the nifi path defined.

Does the Alias name matter to nifi? I saw in the logs where it noticed "client" but that was about it that it recognized. Once I got to the point where it was giving me the cipher mismatches, Nifi errors showed:

NiFi Web Server-51, fatal error: 40: no cipher suites in common

It would say that it is ignoring a large host of possible connection types available. I don't remember the exact ones, but it would say something similar to (this is not the exact code, but similar)

Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256
Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_RC4_128_SHA
Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_RC4_128_SHA
Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_RC4_128_SHA
Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384
Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256
Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256
Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_RC4_128_SHA
Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_RC4_128_SHA

I tried modifying the java.security settings to allow more connections, but it didn't seem to work. I can't tell if I generated the key wrong again or if there is something else I'm missing.

I followed these steps:

keytool -genkey -alias mydomain -keyalg RSA -keystore keystore.jks -keysize 2048

keytool -certreq -alias mydomain -keystore keystore.jks -file mydomain.csr

keytool -import -trustcacerts -alias root -file <signed PEM> -keystore keystore.jks

For the truststore, I simply put all the PEMs into one file and did an import with -trustcacerts.

I attempted this on two different machines and two different browsers, just to make sure I wasn't crazy (that doesn't stop the fact that I might be).

I tried using a TLS authentication tool, and it showed me that TLS1.2 was enabled. I tried openssl and it tried to connect, but it didn't do anything afterwards.

86606-capture.png

I'm still a little lost. Thanks for the help.

avatar

Justin,

I made a diagram to hopefully make this clearer. The desired scenario you are describing -- NiFi secured with a certificate signed by an external, trusted certificate authority -- is definitely supported. You just have to ensure that the correct entities have the correct public and private keys, and that the trust chain is intact.

In this case, you need to have the NiFi public certificate (green) be signed by the Intermediate CA (orange), the client public certificate (blue) signed by the Intermediate CA (orange), and the Intermediate CA public certificate (orange) in the NiFi truststore. That tells NiFi to accept any client certificate that is signed by the Intermediate CA as valid (authorization of the DN to follow after this step). For your browsers to mark the NiFi instance as trusted, the Intermediate CA public certificate (orange) must be imported into the browser's truststore (or the client machine OS keychain, depending on the browser).

This can be confusing. In general, if something possess the private key X, it is X. If it possess the public certificate X, it trusts X.

  1. The client browser needs to possess the Intermediate CA public certificate to trust that NiFi is who it claims it is.
  2. The client browser needs to possess the client private key to tell NiFi it is User Jane Han. The matching client public certificate (what the browser actually sends across the network) needs to be signed by Intermediate CA private key for NiFi to trust that Mallory Gonzalez didn't just make her own certificate that says "I am Jane Han". Similarly, there could be a different Jane Han in a different organization that is not authorized to access NiFi; her (valid) certificate that says "I am Jane Han" isn't signed by your Intermediate CA, so NiFi will reject it.
  3. The NiFi truststore can contain many public certificates. Every public certificate contained there represents a trusted signer (via the corresponding private key). In the scenario below, NiFi will trust any certificate signed by either Intermediate CA or Root CA. If you added "Bernadette's CA" public certificate to that truststore, any certificates signed by that private key would also be accepted by NiFi.
  4. The commands you copied are not the complete series of steps necessary. For one thing, I don't see any reference to truststore.jks at all. I believe you may have also generated the private key under one alias ("mydomain") in the keystore.jks file but then imported the signed public certificate via the certificate signing request to a different alias ("root"). The alias of the signed public certificate and the private key need to be the same. See this Oracle doc for those steps (note the CA public cert is imported under a different alias, but the signed public cert is imported with the original alias).

    86628-untitled-diagram.png

avatar
Contributor

Hey @Justin Brock,

As @Andy LoPresto mentions securing NiFi with an internal CA is 100% possible. In fact Andy's advice helped me significantly when I configured our secure NiFi instance so all the credit goes to him or I'd still be stuck to this date (and he's a top bloke to boot!!!). I detailed the step by step process that I took here securing our NiFi instance using an internal CA (the example uses NiFi 1.3 but it's just as applicable now) so this combined with Andy's detail below is hopefully enough to get you over the line. Best of luck.

avatar

I'm pretty darn sure that your answer is correct @Andy LoPresto and with the help of @D H I should be able to get it. Unfortunately, something strange is still going on. It may have something to do with our internal CA process. I'm going to reach out to our group and see if they can help me out. Something odd is going on.

Thank you for your help!

avatar

Justin,

A couple other things to look at:

  • If your certificate is invalid for any reason (expired, not yet valid, revoked, etc.) it can manifest in a cipher suite mismatch or no cipher suites in common error because any cipher suite that relies on RSA signatures or encryption will be unavailable. Double check that the certificate validity dates are correct and the hostname and SAN match.
  • You can use some open source tools to check the list of cipher suites your server supports -- I recommend cipherscan as that is what I usually use, but TLSSLed is another one I have heard of. I am not as familiar with Windows as Mac OS X/*nix systems, but you can have OpenSSL list the cipher suites available to it (as a client) as well with openssl ciphers -V -tls1 (NiFi 1.2+ only supports incoming TLSv1.2 connections, but there isn't a flag for TLSv1.2 only in OpenSSL yet). If you prefer to do the scanning manually, you can use the script below from https://www.securityevaluators.com/using-openssl-determine-ciphers-enabled-server/.
  • for v in ssl2 ssl3 tls1 tls1_1 tls1_2; do
     for c in $(openssl ciphers 'ALL:eNULL' | tr ':' ' '); do
     openssl s_client -connect <server>:<port> \
     -cipher $c -$v < /dev/null > /dev/null 2>&1 && echo -e "$v:\t$c"
     done
    done
    

avatar

Hi @justin_brock  Believe you were able to fix certificate issue ? I'm have enables ssl for NIFI in CDF but facing "ERR_BAD_SSL_CLIENT_AUTH_CERT" here is link to my question of community 

 

https://community.cloudera.com/t5/Support-Questions/Unable-to-open-NIFI-web-UI-after-TLS/m-p/289190#...

 

could you please help me with steps you followed to resolve issue ?