NiFi Cluster on AWS EC2 With SSL Using tls-toolkit.sh for Certs

Question

I am configuring a new 3 node NiFi cluster on three AWS EC2 instances. I'm attempting to use the tls-toolkit.sh utility to generate the SSL keystore.jks and truststore.jks for each of the 3 nodes. It generates them and I can start the cluster but when I load the UI it seems like it's working but then I get this message.

An unexpected error has occurred (log out) (home)

javax.net.ssl.SSLPeerUnverifiedException: Hostname my.elb.name not verified: certificate: sha256/LhH2H...hkIE= DN: CN=node1.server.name, OU=NIFI subjectAltNames: [node1.server.name]

The nifi-app.log has basically the same to say about it.

WARN [Replicate Request Thread-7] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/current-user to my.elb.name:port due to javax.net.ssl.SSLPeerUnverifiedException: Hostname my.elb.name not verified: ... subjectAltNames: [node1.server.name]

I would have expected the subjectAltNames to include my.elb.name because I included it in my tls-toolkit command.

export JAVA_HOME="/usr/lib/jvm/java-11-openjdk-11.0.19.0-7-4.el8.x86_64"

./tls-toolkit.sh standalone -n 'node1.server.name,node2.server.name,node3.server.name' --subjectAlternativeNames 'CN=my.elb.name, OU=NIFI'

To investigate further I performed the following commands to examine the contents of what tls-toolkit generated for me.

keytool -list -v -keystore keystore.jks -storepass redactedpassword1 -storetype JKS

keytool -list -v -keystore truststore.jks -storepass redactedpassword2 -storetype JKS

The output contains no reference to my.elb.name at all. They only list node1.server.name as a SubjectAlternativeName: DNSName.

I'm guessing that I'm calling tls-toolkit.sh incorrectly but I'm not sure how to do it correctly. I've been using these pages as reference but I guess I'm missing some important detail.

https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#standalone

Generating Knox Certificates Using the TLS Toolkit (cloudera.com)

Can someone please help me understand what I'm doing wrong?

Here are a few variations on the command that I tested but have also have not resulted in the expected Subject Alternative Name in the keytool -list.

(single dash instead of double dash)

./tls-toolkit.sh standalone -n 'node1.server.name,node2.server.name,node3.server.name' -subjectAlternativeNames 'CN=my.elb.name, OU=NIFI'

(my.elb.name without the CN and OU)

./tls-toolkit.sh standalone -n 'node1.server.name,node2.server.name,node3.server.name' --subjectAlternativeNames 'my.elb.name'

(double quotes instead of single quotes for the SAN parameter)

./tls-toolkit.sh standalone -n 'node1.server.name,node2.server.name,node3.server.name' --subjectAlternativeNames "CN=my.elb.name, OU=NIFI"

MattWho · Accepted Answer

@OpenText-Orion

SubjectAlternativeNames would not be full distinquished names (DNs).
SANs are used to verify the within the TLS exchange that the client is connected to correct intended target server. This avoid things like man in the middle attacks.

So essentially what you have are certificate you created for your 3 NiFi nodes:

node1.server.name
node2.server.name
node3.server.name

However, when you are trying to connect to a NiFi node, you are entering https://my.elb.name:<port>/nifi in your browser which is directed to a NiFi node. Without a SAN entry present that matches the target hostname, the TLS exchange assumes the request was not intended for this target host resulting in the exception you see.

Recreate your node certificates using only Hostnames as the SAN entries instead of a full DN.

I executed the following tls-toolkit.sh command you shared and it provided correct expected output:

./tls-toolkit.sh standalone -n 'node1.server.name,node2.server.name,node3.server.name' --subjectAlternativeNames 'my.elb.name'

Alias name: nifi-key
Creation date: Sep 8, 2023
Entry type: PrivateKeyEntry
Certificate chain length: 2
Certificate[1]:
Owner: CN=node1.server.name, OU=NIFI
Issuer: CN=localhost, OU=NIFI
Serial number: 18a76360ce500000000
Valid from: Fri Sep 08 19:12:48 UTC 2023 until: Thu Dec 11 19:12:48 UTC 2025
Certificate fingerprints:
         MD5:  75:70:0C:4F:41:D8:EA:9D:35:46:9E:C1:3B:9C:B0:E9
         SHA1: 5C:0C:CC:B3:C8:29:62:05:5D:5B:C5:BB:71:39:20:40:48:CE:38:A5
         SHA256: 17:79:FF:87:31:07:CB:9A:01:A5:82:03:A4:1B:3F:3D:F0:C3:79:21:C6:90:06:82:3D:FC:A1:0A:5F:64:DB:DE
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: DA A8 38 36 C2 61 E3 CB   DF 66 72 B5 FF D6 B7 F8  ..86.a...fr.....
0010: 92 2B 50 81                                        .+P.
]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:false
  PathLen: undefined
]

#3: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  clientAuth
  serverAuth
]

#4: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Non_repudiation
  Key_Encipherment
  Data_Encipherment
  Key_Agreement
]

#5: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: node1.server.name
  DNSName: my.elb.name
]

#6: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 05 52 D3 51 9B 56 27 EB   D2 C1 62 42 A9 43 39 EF  .R.Q.V'...bB.C9.
0010: 3A 8E 0D 42                                        :..B
]
]

Make sure you are looking at the PrivateKeyEntry certificate [1]. certificate[2] in the PrivateKeyEntry is the signing certificate.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

OpenText-Orion · Answer

Thank you @MattWho for the answer and @cjervis for the tags.

That did solve my issue but it took me a couple tries. I will be marking yours as the solution but including this note for future readers in case they run into the same thing I did. I had been copy-pasting commands in order to keep notes on my steps and at some point one of the two dashes on the --subjectAlternativeNames had been replaced by a different kind of dash character.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.