SSLError CertificateError - hostname 'dev.example.com' doesn't match 'nifi-node1'

[#1]

I'm running 3 node docker cluster on a single host lets say `dev.example.com`

I have followed https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html#creating-and-securing-a-nifi-cluster-w...  but added every node into a seperate docker container

nifi-node1 0.0.0.0:8096 -> 8080/tcp
nifi-node2 0.0.0.0:8097 -> 8080/tcp
nifi-node3 0.0.0.0:8098 -> 8080/tcp

 
I'm accessing using https://dev.example.com:8096/nifi . I have imported client cert on my browser, I'm able to access everything from Chrome.

I want to use python to start/stop/manipulate the processors using nipyapi
 

import os
import nipyapi

SSL_DIR = '/home/ubuntu/ssl'
nipyapi.config.nifi_config.host = 'https://dev.example.com:8096/nifi-api'

ca_file = os.path.join(SSL_DIR, 'nifi-cert.pem')
client_cert_file = os.path.join(SSL_DIR, 'CN=nifi-node1_OU=NIFI.pem')
client_key_file = os.path.join(SSL_DIR, 'CN=nifi-node1_OU=NIFI_key.pem')
client_key_password = 'Random_Password'

nipyapi.security.set_service_ssl_context(service='nifi', ca_file=ca_file, client_cert_file=client_cert_file, client_key_file=client_key_file, client_key_password=client_key_password)

nipyapi.canvas.get_root_pg_id()

​
 
I'm receiving following error
 

Spoiler

WARNING:urllib3.connection:Certificate did not match expected hostname: dev.example.com. Certificate: {'subject': ((('organizationalUnitName', 'NIFI'),), (('commonName', 'nifi-node1'),)), 'issuer': ((('organizationalUnitName', 'NIFI'),), (('commonName', 'localhost'),)), 'version': 3, 'serialNumber': '017745358C8500000000', 'notBefore': 'Jan 27 18:57:53 2021 GMT', 'notAfter': 'May 2 18:57:53 2023 GMT', 'subjectAltName': (('DNS', 'nifi-node1'),)}

MaxRetryError: HTTPSConnectionPool(host='dev.example.com', port=8096): Max retries exceeded with url: /nifi-api/flow/process-groups/root/status (Caused by SSLError(CertificateError("hostname 'dev.example.com' doesn't match 'nifi-node1'",),))

[#2 · Accepted solution]

Hello @BhaveshP 

The error you are seeing is because certificate CN/SAN name mismatch than that of hostname. Let me try to explain you with an example.

client [A]--> Balancer[B] -->NIFI Nodes[C,D,E]

Here you have a client A who wants to access NIFI nodes C,D and E and he wants to access via B. When you create SSL certificates for C,D,E, you create 3 certificates with 3 different "Common Names" C, D. E respectively. Now when you try connecting to nifi nodes C,D,E respectively directly from client A, you will not observe the issue. But when you try to access C,D,E via balancer or any proxy B, you are likely to get the error.

WHY?

Client A is trying to talk to Nifi Node C or D or E but to client the NIFI node is B. During SSL handshake, the certificates are passed by Nifi server to A where in client gets confused becuase it wanted to talk to B however the certificate it got is from C. Name don't match.

FIX: Make use to SAN Names in certificates. (Subject Alternative Names)

Issue certificate of C,D,E such that it has a SAN name as B. What I am trying to say is, the certificate should have both the name that of C and B in nifi node C, D and B in nifi node D and E and B in nifi node E. In this way client will come to know that when it talks to B and when the certificate received to A from C, it will not get confused because the certificate will have two name present in SAN that of B and C. 

Let me know if the above gives some clarity as to what exactly is happening.

[#3 · Accepted solution]

Hello @BhaveshP 

The error you are seeing is because certificate CN/SAN name mismatch than that of hostname. Let me try to explain you with an example.

client [A]--> Balancer[B] -->NIFI Nodes[C,D,E]

Here you have a client A who wants to access NIFI nodes C,D and E and he wants to access via B. When you create SSL certificates for C,D,E, you create 3 certificates with 3 different "Common Names" C, D. E respectively. Now when you try connecting to nifi nodes C,D,E respectively directly from client A, you will not observe the issue. But when you try to access C,D,E via balancer or any proxy B, you are likely to get the error.

WHY?

Client A is trying to talk to Nifi Node C or D or E but to client the NIFI node is B. During SSL handshake, the certificates are passed by Nifi server to A where in client gets confused becuase it wanted to talk to B however the certificate it got is from C. Name don't match.

FIX: Make use to SAN Names in certificates. (Subject Alternative Names)

Issue certificate of C,D,E such that it has a SAN name as B. What I am trying to say is, the certificate should have both the name that of C and B in nifi node C, D and B in nifi node D and E and B in nifi node E. In this way client will come to know that when it talks to B and when the certificate received to A from C, it will not get confused because the certificate will have two name present in SAN that of B and C. 

Let me know if the above gives some clarity as to what exactly is happening.

[#4 · Accepted solution]

@BhaveshP 

I am in complete agreement with @tusharkathpal response.

But you should be able to work around this issue through a configuration change in your nifi.properties file.

nifi.web.proxy.host=dev.example.com:<port number>

Property description: A comma separated list of allowed HTTP Host header values to consider when NiFi is running securely and will be receiving requests to a different host[:port] than it is bound to. For example, when running in a Docker container or behind a proxy (e.g. localhost:18443, proxyhost:443). By default, this value is blank meaning NiFi should only allow requests sent to the host[:port] that NiFi is bound to.

Since the hostname your client is using does not match any SAN in the individual nodes certificates, the above property allows NiFi to accept this additional hostname.

The other option is to create new certificates for each of your NiFi nodes where "dev.example.com' is added as an additional SAN entry.

Hope this helps,

Matt

[#5]

@tusharkathpal @MattWho 

Thanks, Guys for the help.

I have tried it out the same using following command

```

./tls-toolkit.sh standalone -n 'nifi-node1,nifi-node2,nifi-node3' -C 'CN=admin, OU=NIFI' --nifiDnPrefix 'CN=' --nifiDnSuffix ', C=IN' -K randompassword -P randompassword -S randompassword -B randompassword -o /tmp/5/certs/ssl --subjectAlternativeNames 'dev.example.com'

```

While deployment, It's giving me following error and killing the containers

2021-02-03 07:52:41,921 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead)

-----------------  Update -----------------

Above issue was introduced in the latest version (1.12.x):

https://issues.apache.org/jira/browse/NIFI-7730

Issue resolved by adding 'dev.example.com' as an additional SAN entry and by upgrading the NiFi version to nifi-1.13.0 (dev)