insufficient permissions untrusted proxy

[#1]

I have step up 3 node (node1, node2, node3) nifi cluster and I am able to access it with http protocol, When I try to make those 3 node secure(https). I am facing "insufficient permissions untrusted proxy" in web UI.

After setup 3 node cluster in node1 i followed the steps to make secure connection.

 

Step:1 By using below step I have Generate Keys and Certificates

/opt/nifi-toolkit/bin/tls-toolkit.sh standalone -n 'node1' -B ClientPassword -C 'CN=admin,OU=NIFI' -O -o /opt/nifi-1.19.1/conf/certs

after run this I got below files

/opt/nifi-1.19.1/conf/certs/
├── CN=admin_OU=NIFI.p12
├── CN=admin_OU=NIFI.password
├── node1
│ ├── keystore.jks
│ ├── nifi.properties
│ └── truststore.jks
├── nifi-cert.pem
└── nifi-key.key

Step:2

I have over write the /opt/nifi-1.19.1/conf/certs/node1/nifi.properties to conf/nifi.properties

Then Edit the nifi.properties

nifi.security.user.authorizer=managed-authorizer

Step:3

Updated Security Properties Key

/opt/nifi-1.19.1/bin/nifi.sh set-sensitive-properties-key xyz12345

Step:4 Edited the authorizers.xml as below
<authorizers>
<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<property name="Initial User Identity 1">CN=admin, OU=NIFI</property>
</userGroupProvider>

<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">CN=admin, OU=NIFI</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1">CN=admin, OU=NIFI</property>
<property name="Node Group"></property>
</accessPolicyProvider>

<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers>

Step:4 Removed users.xml authorizations.xml

cd ./conf/
rm -rf users.xml authorizations.xml

Step:5 started the nifi
./nifi.sh start
Imported the CN=admin_OU=NIFI.p12 certs to my browser and try to accessing the I am able to access it, after that I have noted in nifi.properties nifi.cluster.is.node=false then i changed to nifi.properties nifi.cluster.is.node=ture

after changing to true I am unable to access it. getting error in web UI like  insufficient permissions untrusted proxy CN=node1, OU=NIFI

[#2 · Accepted solution]

@apmmahesh 
You created certificates for each of your NiFi nodes.  Base on exception you shared, it appears that you created DNs for those nodes as following?
CN=node1, OU=NIFI
CN=node2, OU=NIFI
CN=node3, OU=NIFI

When you have a NiFi cluster, you can manage that cluster via the UI of any one of the connected nodes.
So let's say you authenticate via a mutual TLS handshake to node1 using your CN=admin, OU=NIFI certificate you created for yourself and loaded in your browser.  What happens next is node1 wants to show you all the data/details from all three nodes and not just node1, so your request to load the NiFi is sent via proxy by node1 to whichever node is the elected cluster coordinator.  That cluster coordinator replicates the request on your behalf to all nodes in the cluster.  This how the node1 UI would show you details about connected nodes, queued data from other nodes, etc.  This means that node1 would need to be authorized to proxy user requests.  So typically on first startup secure NiFi will use the configuration in your authorizers.xml to setup these needed default authorization, but your configuration is missing your nodes, so this was not done.

Inside your file user-group-provider, you need to also add your NiFi node DNs as users.

<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<property name="Initial User Identity 1">CN=admin, OU=NIFI</property>
<property name="Initial User Identity 2">CN=node1, OU=NIFI</property>
<property name="Initial User Identity 3">CN=node2, OU=NIFI</property>
<property name="Initial User Identity 4">CN=node3, OU=NIFI</property>
</userGroupProvider>

Then in your file-access-policy-provider you need to add your nodes so that when it generates the authorizations.xml file, the nodes get authorized to the "proxy user requests" policy:

<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">CN=admin, OU=NIFI</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1">CN=node1, OU=NIFI</property>
<property name="Node Identity 2">CN=node2, OU=NIFI</property>
<property name="Node Identity 3">CN=node3, OU=NIFI</property>
<property name="Node Group"></property>
</accessPolicyProvider>

NOTE:  NiFI will only create the users.xml and authorizations.xml files from the above two providers if they do NOT already exist.  Making changes to these providers will not result in changes to existing files.  The expectation is that after access for yoru initial admin and your proxy nodes is established that all new authorizations are setup via the NiFi UI which will result in updated to these files.  So rename your existing users.xml and authorizations.xml before starting yoru NiFi so new get created.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

[#3]

Thanks @MattWho  after adding that I am getting new error now in web UI

1.When I try to access node1 in web UI I am getting "Invalid State The Flow Controller is initializing The Data Flow"

2.When I try to access node2 in web UI I am able to access connected/Total number of nodes in the cluster is 1/3

3.When I try to access node3 in web UI I am getting "Invalid State The Flow Controller is initializing The Data Flow"

node1 nifi-app.log 

2023-03-21 17:53:41,519 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at Node2:11443; will use this address for sending heartbeat messages
2023-03-21 17:53:41,519 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at Node2:11443. Will send Cluster Connection Request to this address
2023-03-21 17:53:41,522 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
node2 nifi-app.log 
WARN [Process Cluster Protocol Request-26] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from node1 due to Received fatal alert: certificate_unknown

WARN [Process Cluster Protocol Request-26] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from node3 due to Received fatal alert: certificate_unknown

node3 nifi-app.log 

2023-03-21 17:53:41,519 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at Node2:11443; will use this address for sending heartbeat messages
2023-03-21 17:53:41,519 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at Node2:11443. Will send Cluster Connection Request to this address
2023-03-21 17:53:41,522 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

node1 verbose output of NiFi keystore # keytool -v -list -keystore keystore.jks

Owner: CN=nifi1, OU=NIFI
Issuer: CN=localhost, OU=NIFI
Serial number: 186e9f79f1000000000
Valid from: Thu Mar 16 18:29:24 CST 2023 until: Wed Jun 18 18:29:24 CST 2025
Certificate fingerprints:
MD5: 69:05:14:4A:61:5A:4A:E4:36:2F:8B:C9:E8:C5:D7:EC
SHA1: 1F:98:8A:64:C4:DF:C0:E4:A4:8E:9A:F8:44:9A:DD:24:87:B1:54:A3
SHA256: FA:A8:A0:DC:F3:15:41:28:26:D9:88:70:E9:B8:C9:0F:37:BC:C8:3F:00:E0:A2:D4:6C:55:74:C1:07:90:14:DF
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 8B 2C 24 76 04 D0 83 C8 25 36 12 0D D1 4D A0 57 .,$v....%6...M.W
0010: F9 18 60 BD ..`.
]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:false
PathLen: undefined
]

#3: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
clientAuth
serverAuth
]

#4: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Non_repudiation
Key_Encipherment
Data_Encipherment
Key_Agreement
]

#5: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: nifi1
]

#6: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 33 65 62 1D FB EE 28 01 65 57 F0 12 15 B2 8C 89 3eb...(.eW......
0010: 4B FE A8 A4 K...
]
]

Certificate[2]:
Owner: CN=localhost, OU=NIFI
Issuer: CN=localhost, OU=NIFI
Serial number: 186e9f79e9200000000
Valid from: Thu Mar 16 18:29:24 CST 2023 until: Wed Jun 18 18:29:24 CST 2025
Certificate fingerprints:
MD5: F3:BC:5B:ED:1D:AA:BC:23:17:2E:C1:FB:A5:1F:F6:66
SHA1: AE:BF:55:52:B3:75:24:81:4F:0E:F5:22:62:E9:00:2C:10:87:1A:40
SHA256: 1E:9A:AB:40:E7:C8:82:0B:11:7A:95:3F:73:09:00:29:38:50:59:61:44:BC:3D:0A:42:C2:1F:FC:D0:D7:63:8A
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 8B 2C 24 76 04 D0 83 C8 25 36 12 0D D1 4D A0 57 .,$v....%6...M.W
0010: F9 18 60 BD ..`.
]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:true
PathLen:2147483647
]

#3: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
clientAuth
serverAuth
]

#4: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Non_repudiation
Key_Encipherment
Data_Encipherment
Key_Agreement
Key_CertSign
Crl_Sign
]

#5: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: localhost
]

#6: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 8B 2C 24 76 04 D0 83 C8 25 36 12 0D D1 4D A0 57 .,$v....%6...M.W
0010: F9 18 60 BD ..`.
]
]

*******************************************
*******************************************

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore keystore.jks -destkeystore keystore.jks -deststoretype pkcs12".

Nifi version: 1.19.1

java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)

I am very new to nifi. Let me know if you need any other information 

[#4]

@apmmahesh 
Your current issue is unrelated to your original issue.  What you have now is an issue with your mutual TLS exchange between your cluster nodes. NiFi does nothing unusual or out of spec with regards to TLS.

Specifically this line snippet you shared:

javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

points to a trust issue within the exchange.

So few things about your certificates:
1. Is "nifi1" the actual hostname for this server? (same for nifi2 and nifi3)
2. What is inside the truststore.jks you are using on each of your NiFi nodes?  The complete TLS trustchain for your NiFi certificates must be in this truststore.jks.  Your truststore.jks should contain one too many TrustedCertEntries.  Your keystore.jks must contain only 1 PrivateKeyEntry.

Assuming that all your node's certificates were generated using the NiFi TLS toolkit, they should all be issued/signed by the same certifcate Authority (Issuer: CN=localhost, OU=NIFI).  Their then must be a a trustedCertEntry present in the trustsore.jks used by all your nodes for "CN=localhost, OU=NIFI".

You can use the same keytool -v -list -keystore truststore.jks command to verify the contents of your truststore.jks.

You can also use the following command to obtain the public cert(s) for your CA from any one of your NiFi servers:

openssl s_client -connect <nifi hostname>:<nifi port> -showcerts

each public cert will have this structure:

-----BEGIN CERTIFICATE-----
MIIFYjCCBEqgAw..............b9Sz/yilrbCgj8=
-----END CERTIFICATE----

I cut out portion of string in above, but the certificate includes everything above in it.  You can copy each and save them to "<filename>.crt" files which you can import to your truststore.jks files (all nodes shoudl e using a common truststore.jks) if they are missing.

keytool -import -alias <some unique ca name> -file <filename>.crt -keystore truststore.jks -storepass <truststore.jks password>

You'll need to restart your NiFi anytime you modify either the keystore or truststore.  NiFi only reads these files on startup.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

[#5]

@MattWho Thanks for the support. I have added the all 3 node public cert(s) in single file (nifi.crt).

After that I run below command in each node.

keytool -import -alias devnifi -file nifi.crt -keystore truststore.jks -storepass xyz

For your information
openssl s_client -connect nifi1.net:9443 -showcerts output of node1

CONNECTED(00000003)
depth=1 OU = NIFI, CN = localhost
verify error:num=19:self signed certificate in certificate chain
140139985950608:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:s3_pkt.c:1493:SSL alert number 42
140139985950608:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
 0 s:/OU=NIFI/CN=nifi1.net
   i:/OU=NIFI/CN=localhost
-----BEGIN CERTIFICATE-----
MIIDjDCCAnSgAwIBAgIKAYbp958QAAAAADANBgkqhkiG9w0BAQsFADAjMQ0wCwYD
VQQLDAROSUZJMRIwEAYDVQQDDAlsb2NhbGhvc3QwHhcNMjMwMzE2MTAyOTI0WhcN
MjUwNjE4MTAyOTI0WjA6MQ0wCwYDVQQLDAROSUZJMSkwJwYDVQQDDCB0YW9vZHNo
ZHBkbjAxLmxpLmx1bWVudHVtaW5jLm5ldDCCASIwDQYJKoZIhvcoAQEBBQADggEP
ADCCAQoCggEBALXDym1yZ9EiUU30yXp0VmDJvujqcnBnEVqQBVIWXOgBQxbol/Pq
5J7T3+zd4QPbIggsDeJyZG4kDeWUEbNFv1yiy+zR9atUj8DwRrNPSvgn+6U9DQSp
1EuEr35IA0VUJNuoPapvCeeOnqHiS7HR92XgrfgOxEW3Qj/phnQAEEz5BYn1RlYn
k7HklgqOIjkRd7HVZ8JIQEFMUChAzRwdJxa3ty+gMsebGP0K8BWFCeL3DEBlerdj
agapX91yhTayDT06y59IVQ98rIRs6TDbL/vJpTlbaJqo/UH4EtBUyy8fy9vIHTPF
4aKgNf23VoN6EqtBSb6y8Qf3wHjoPoQpF1cCAwEAAaOBqjCBpzAdBgNVHQ4EFgQU
M2ViHfvuKAFlV/ASFbKMiUv+qKQwHwYDVR0jBBgwFoAUiywkdgTQg8glNhIN0U2g
V/kYYL0wDgYDVR0PAQH/BAQDAgP4MAkGA1UdEwQCMAAwHQYDVR0lBBYwFAYIKwYB
BQUHAwIGCCsGAQUFBwMBMCsGA1UdEQQkMCKCIHRhb29kc2hkcGRuMDEubGkubHVt
ZW50dW1pbmMubmV0MA0GCSqGSIb3DQEBCwUAA4IBAQBvhTwotQ2N14q7+mfnFE/g
RMdQCoMC3Ww+C1/v/ZpMykst4YQAbW0BXPP3wofy604L12G1IhfLiwGGBc+K89Q/
4HfDsFtQ30g4Ovier3NdwTlLihgHJj36s9ykPox/DSr8oQeKBPFy+++XVLfbtoVz
GO7HpQMXUwEa2RCpFjDzmAQd2Cc6QjtB1gZ6tT3bjAHHEy963V+LbD1YFSLqNWGw
r2bkVcRVuCy+jkwFfNn4ZO2U/OYNIfrOiSrektMwbExtgGOKn/drKC3aELeWOmfR
SeHy5kfxHrFPv2aP+8Dcg61Bj5HQk3B0A9SLRUyJeK7v/XnNypH56oejy6al/Tu5
-----END CERTIFICATE-----
 1 s:/OU=NIFI/CN=localhost
   i:/OU=NIFI/CN=localhost
-----BEGIN CERTIFICATE-----
MIIDYTCCAkmgAwIBAgIKAYbp956SAAAAADANBgkqhkiG9w0BAQsFADAjMQ0wCwYD
VQQLDAROSUZJMRIwEAYDVQQDDAlsb2NhbGhvc3QwHhcNMjMwMzE2MTAyOTI0WhcN
MjUwNjE4MTAyOTI0WjAjMQ0wCwYDVQQLDAROSUZJMRIwEAYDVQQDDAlsb2NhbGhv
c3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/yg1UnKfLvj2LgJ11
VGeF58CZcR/3DSztOVeBuy+eNHIt9so5D7OWlcMkZzEyxurY4e2gHBNvZowhSK2l
VO/H4b4rE4FUtmpLu68fXZWgioENF2xQ6xeisVxQuxEvsY6dl7DkyBAvp8zKiuQS
3momXYZeEDDHXfDx1v0lDSntlQNe4Fb88ROweOqc3CMtyEs/9fhjW/iAFzCmcqC4
0uvSJyCYJGyMspQ/mRU4SaMqqe0WJ3Je/wwIXxqbdMGksavUCa976OEk0rqPxvf/
pw6armSwKuZVICt8WGKoQw2rbwEo4jIeU8DdhAcM2GhTdgEJ3WCs+F2gu20meEHw
a6htAgMBAAGjgZYwgZMwDgYDVR0PAQH/BAQDAgH+MAwGA1UdEwQFMAMBAf8wHQYD
VR0OBBYEFIssJHYE0IPIJTYSDdFNoFf5GGC9MB8GA1UdIwQYMBaAFIssJHYE0IPI
JTYSDdFNoFf5GGC9MB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATAUBgNV
HREEDTALgglsb2NhbGhvc3QwDQYJKoZIhvcNAQELBQADggEBAKpXb5lPlu8iaQI1
RFGQHjcoGJKRt8C3EHqvEbXeLT1Zck1NAKWsM+YYveX4oLanS5gMPZbcGm7sCaxk
BkdUP+mMej3+K6fiViexztn2ZRW4u3gV4/dVx/li0Z7Aw1l+Ebm6zSWxujnzJwOP
BjveQHi7JYj6g4Axb82GUOFyjJraX796sUKiNjeDDBr/mh1nImrGrNgiNzR+a5Ri
AV4MN9dfQs23fTkPo3YwcbAt1vTphco6owUEwtf7NB+re7fs0rZ+iq3o+d2fGN9W
+oRJeYeMdesaHaUR6U1SqYyeZ/TNrkRbGuBF79xXBv7azXyZPxNj8Q6iPQ/Pc6H5
sC6YSao=
-----END CERTIFICATE-----
---
Server certificate
subject=/OU=NIFI/CN=nifi1.net
issuer=/OU=NIFI/CN=localhost
---
Acceptable client certificate CA names
/OU=NIFI/CN=nifi1.net
/OU=NIFI/CN=localhost
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2363 bytes and written 194 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-SHA384
    Session-ID: 6421586CFF9821F1E9851E6E7517D0DB5599F0A20A021033BF64B469610E5D50
    Session-ID-ctx:
    Master-Key: 244C473DE1585F13FFCA6682898903ED4A4287F1A1921D705437DDB7660847F54032543A24D474D14487867FC67BF895
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1679906924
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---

keytool -v -list -keystore truststore.jks output of node1

Enter keystore password:
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 2 entries

Alias name: devnifi
Creation date: Mar 27, 2023
Entry type: trustedCertEntry

Owner: CN=nifi1.net, OU=NIFI
Issuer: CN=localhost, OU=NIFI
Serial number: 186e9f79f1000000000
Valid from: Thu Mar 16 18:29:24 CST 2023 until: Wed Jun 18 18:29:24 CST 2025
Certificate fingerprints:
         SHA1: 1F:98:8A:64:C4:DF:C0:E4:A4:8E:9A:F8:44:9A:DD:24:87:B1:54:A3
         SHA256: FA:A8:A0:DC:F3:15:41:28:26:D9:88:70:E9:B8:C9:0F:37:BC:C8:3F:00:E0:A2:D4:6C:55:74:C1:07:90:14:DF
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 8B 2C 24 76 04 D0 83 C8   25 36 12 0D D1 4D A0 57  .,$v....%6...M.W
0010: F9 18 60 BD                                        ..`.
]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:false
  PathLen: undefined
]

#3: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  clientAuth
  serverAuth
]

#4: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Non_repudiation
  Key_Encipherment
  Data_Encipherment
  Key_Agreement
]

#5: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: nifi1.net
]

#6: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 33 65 62 1D FB EE 28 01   65 57 F0 12 15 B2 8C 89  3eb...(.eW......
0010: 4B FE A8 A4                                        K...
]
]



*******************************************
*******************************************


Alias name: nifi-cert
Creation date: Mar 16, 2023
Entry type: trustedCertEntry

Owner: CN=localhost, OU=NIFI
Issuer: CN=localhost, OU=NIFI
Serial number: 186e9f79e9200000000
Valid from: Thu Mar 16 18:29:24 CST 2023 until: Wed Jun 18 18:29:24 CST 2025
Certificate fingerprints:
         SHA1: AE:BF:55:52:B3:75:24:81:4F:0E:F5:22:62:E9:00:2C:10:87:1A:40
         SHA256: 1E:9A:AB:40:E7:C8:82:0B:11:7A:95:3F:73:09:00:29:38:50:59:61:44:BC:3D:0A:42:C2:1F:FC:D0:D7:63:8A
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 8B 2C 24 76 04 D0 83 C8   25 36 12 0D D1 4D A0 57  .,$v....%6...M.W
0010: F9 18 60 BD                                        ..`.
]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:true
  PathLen:2147483647
]

#3: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  clientAuth
  serverAuth
]

#4: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Non_repudiation
  Key_Encipherment
  Data_Encipherment
  Key_Agreement
  Key_CertSign
  Crl_Sign
]

#5: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: localhost
]

#6: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 8B 2C 24 76 04 D0 83 C8   25 36 12 0D D1 4D A0 57  .,$v....%6...M.W
0010: F9 18 60 BD                                        ..`.
]
]



*******************************************
*******************************************

Then I have delete the all 3 node authorizations.xml and users.xml file and restarted the  nifi.sh service and zookeeper service as well.
same issue in web UI

1.When I try to access node1 in web UI I am getting "Invalid State The Flow Controller is initializing The Data Flow"

2.When I try to access node2 in web UI I am getting "Invalid State The Flow Controller is initializing The Data Flow"

3.When I try to access node3 in web UI I am able to access connected/Total number of nodes in the cluster is 1/3

Below are the error log files for all 3 nodes

node1 nifi-app.log

2023-03-27 16:25:26,523 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at node3.net:11443; will use this address for sending heartbeat messages
2023-03-27 16:25:26,523 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at node3.net:11443. Will send Cluster Connection Request to this address
2023-03-27 16:25:26,527 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors


node2 nifi-app.log
2023-03-27 16:23:50,043 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi3.net:11443; will use this address for sending heartbeat messages
2023-03-27 16:23:50,043 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at nifi3.net:11443. Will send Cluster Connection Request to this address
2023-03-27 16:23:50,047 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

node3 nifi-app.log

6:20:11,079 WARN [Process Cluster Protocol Request-25] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from nifi1.net due to Received fatal alert: certificate_unknown
2023-03-27 16:20:11,983 INFO [Process Cluster Protocol Request-26] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 2c0d0d2a-b799-4e40-963e-59adb2e6f2d0 (type=HEARTBEAT, length=4315 bytes) from nifi3.net:9443 in 92 millis
2023-03-27 16:20:11,985 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2023-03-27 16:20:11,888 and sent to nifi3.net:11443 at 2023-03-27 16:20:11,985; determining Cluster Coordinator took 1 millis; DNS lookup for coordinator took 0 millis; connecting to coordinator took 1 millis; sending heartbeat took 51 millis; receiving first byte from response took 42 millis; receiving full response took 43 millis; total time was 97 millis
2023-03-27 16:20:14,820 WARN [Process Cluster Protocol Request-27] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from nifi2.net due to java.net.SocketException: Broken pipe (Write failed)
java.net.SocketException: Broken pipe (Write failed)
at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
at java.base/sun.security.ssl.SSLSocketOutputRecord.flush(SSLSocketOutputRecord.java:271)
at java.base/sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:89)
at java.base/sun.security.ssl.Finished$T13FinishedProducer.onProduceFinished(Finished.java:773)
at java.base/sun.security.ssl.Finished$T13FinishedProducer.produce(Finished.java:661)
at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)
at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.goServerHello(ClientHello.java:1223)
at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(ClientHello.java:1159)
at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:841)
at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:802)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1507)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1417)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:922)
at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1013)
at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:973)
at org.apache.nifi.stream.io.ByteCountingInputStream.read(ByteCountingInputStream.java:42)
at java.base/java.io.FilterInputStream.read(FilterInputStream.java:83)
at org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$2.unmarshal(JaxbProtocolContext.java:113)
at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:150)
at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:131)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-03-27 16:20:15,996 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 83047 nanos
2023-03-27 16:20:16,089 WARN [Process Cluster Protocol Request-28] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from nifi1.net due to java.net.SocketException: Broken pipe (Write failed)
java.net.SocketException: Broken pipe (Write failed)
at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
at java.base/sun.security.ssl.SSLSocketOutputRecord.flush(SSLSocketOutputRecord.java:271)
at java.base/sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:89)
at java.base/sun.security.ssl.CertificateRequest$T12CertificateRequestProducer.produce(CertificateRequest.java:636)
at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)
at java.base/sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1091)
at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:843)
at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:802)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1507)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1417)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:922)
at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1013)
at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:973)
at org.apache.nifi.stream.io.ByteCountingInputStream.read(ByteCountingInputStream.java:42)
at java.base/java.io.FilterInputStream.read(FilterInputStream.java:83)
at org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$2.unmarshal(JaxbProtocolContext.java:113)
at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:150)
at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:131)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

Regards

Mahesh

[#6]

@apmmahesh 

Make sure that the nifi.properties file on all nodes is configured the same.
Make sure that the "nifi.cluster.protocol.is.secure" property is set to true on all the nodes.

Matt