Skip to main content
apmmahesh
apmmahesh
Silver ⭐️⭐️
March 20, 2023
Solved

insufficient permissions untrusted proxy

I have step up 3 node (node1, node2, node3) nifi cluster and I am able to access it with http protocol, When I try to make those 3 node secure(https). I am facing "insufficient permissions untrusted proxy" in web UI.

After setup 3 node cluster in node1 i followed the steps to make secure connection.

 

Step:1 By using below step I have Generate Keys and Certificates

/opt/nifi-toolkit/bin/tls-toolkit.sh standalone -n 'node1' -B ClientPassword -C 'CN=admin,OU=NIFI' -O -o /opt/nifi-1.19.1/conf/certs

after run this I got below files

/opt/nifi-1.19.1/conf/certs/
├── CN=admin_OU=NIFI.p12
├── CN=admin_OU=NIFI.password
├── node1
│ ├── keystore.jks
│ ├── nifi.properties
│ └── truststore.jks
├── nifi-cert.pem
└── nifi-key.key

Step:2

I have over write the /opt/nifi-1.19.1/conf/certs/node1/nifi.properties to conf/nifi.properties

Then Edit the nifi.properties

nifi.security.user.authorizer=managed-authorizer

Step:3

Updated Security Properties Key

/opt/nifi-1.19.1/bin/nifi.sh set-sensitive-properties-key xyz12345

Step:4 Edited the authorizers.xml as below
<authorizers>
<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<property name="Initial User Identity 1">CN=admin, OU=NIFI</property>
</userGroupProvider>

<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">CN=admin, OU=NIFI</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1">CN=admin, OU=NIFI</property>
<property name="Node Group"></property>
</accessPolicyProvider>

<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers>

Step:4 Removed users.xml authorizations.xml

cd ./conf/
rm -rf users.xml authorizations.xml

Step:5 started the nifi
./nifi.sh start
Imported the CN=admin_OU=NIFI.p12 certs to my browser and try to accessing the I am able to access it, after that I have noted in nifi.properties nifi.cluster.is.node=false then i changed to nifi.properties nifi.cluster.is.node=ture

after changing to true I am unable to access it. getting error in web UI like  insufficient permissions untrusted proxy CN=node1, OU=NIFI

MattWho
MattWhoAnswer
Titanium ⭐️⭐️⭐️
March 20, 2023

@apmmahesh 
You created certificates for each of your NiFi nodes.  Base on exception you shared, it appears that you created DNs for those nodes as following?
CN=node1, OU=NIFI
CN=node2, OU=NIFI
CN=node3, OU=NIFI

When you have a NiFi cluster, you can manage that cluster via the UI of any one of the connected nodes.
So let's say you authenticate via a mutual TLS handshake to node1 using your CN=admin, OU=NIFI certificate you created for yourself and loaded in your browser.  What happens next is node1 wants to show you all the data/details from all three nodes and not just node1, so your request to load the NiFi is sent via proxy by node1 to whichever node is the elected cluster coordinator.  That cluster coordinator replicates the request on your behalf to all nodes in the cluster.  This how the node1 UI would show you details about connected nodes, queued data from other nodes, etc.  This means that node1 would need to be authorized to proxy user requests.  So typically on first startup secure NiFi will use the configuration in your authorizers.xml to setup these needed default authorization, but your configuration is missing your nodes, so this was not done.

Inside your file user-group-provider, you need to also add your NiFi node DNs as users.

<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<property name="Initial User Identity 1">CN=admin, OU=NIFI</property>
<property name="Initial User Identity 2">CN=node1, OU=NIFI</property>
<property name="Initial User Identity 3">CN=node2, OU=NIFI</property>
<property name="Initial User Identity 4">CN=node3, OU=NIFI</property>
</userGroupProvider>


Then in your file-access-policy-provider you need to add your nodes so that when it generates the authorizations.xml file, the nodes get authorized to the "proxy user requests" policy:

<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">CN=admin, OU=NIFI</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1">CN=node1, OU=NIFI</property>
<property name="Node Identity 2">CN=node2, OU=NIFI</property>
<property name="Node Identity 3">CN=node3, OU=NIFI</property>
<property name="Node Group"></property>
</accessPolicyProvider>


NOTE:  NiFI will only create the users.xml and authorizations.xml files from the above two providers if they do NOT already exist.  Making changes to these providers will not result in changes to existing files.  The expectation is that after access for yoru initial admin and your proxy nodes is established that all new authorizations are setup via the NiFi UI which will result in updated to these files.  So rename your existing users.xml and authorizations.xml before starting yoru NiFi so new get created.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

 

View in thread

5 replies

MattWho
Titanium ⭐️⭐️⭐️
MattWhoAnswer
Titanium ⭐️⭐️⭐️
March 20, 2023

@apmmahesh 
You created certificates for each of your NiFi nodes.  Base on exception you shared, it appears that you created DNs for those nodes as following?
CN=node1, OU=NIFI
CN=node2, OU=NIFI
CN=node3, OU=NIFI

When you have a NiFi cluster, you can manage that cluster via the UI of any one of the connected nodes.
So let's say you authenticate via a mutual TLS handshake to node1 using your CN=admin, OU=NIFI certificate you created for yourself and loaded in your browser.  What happens next is node1 wants to show you all the data/details from all three nodes and not just node1, so your request to load the NiFi is sent via proxy by node1 to whichever node is the elected cluster coordinator.  That cluster coordinator replicates the request on your behalf to all nodes in the cluster.  This how the node1 UI would show you details about connected nodes, queued data from other nodes, etc.  This means that node1 would need to be authorized to proxy user requests.  So typically on first startup secure NiFi will use the configuration in your authorizers.xml to setup these needed default authorization, but your configuration is missing your nodes, so this was not done.

Inside your file user-group-provider, you need to also add your NiFi node DNs as users.

<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<property name="Initial User Identity 1">CN=admin, OU=NIFI</property>
<property name="Initial User Identity 2">CN=node1, OU=NIFI</property>
<property name="Initial User Identity 3">CN=node2, OU=NIFI</property>
<property name="Initial User Identity 4">CN=node3, OU=NIFI</property>
</userGroupProvider>


Then in your file-access-policy-provider you need to add your nodes so that when it generates the authorizations.xml file, the nodes get authorized to the "proxy user requests" policy:

<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">CN=admin, OU=NIFI</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1">CN=node1, OU=NIFI</property>
<property name="Node Identity 2">CN=node2, OU=NIFI</property>
<property name="Node Identity 3">CN=node3, OU=NIFI</property>
<property name="Node Group"></property>
</accessPolicyProvider>


NOTE:  NiFI will only create the users.xml and authorizations.xml files from the above two providers if they do NOT already exist.  Making changes to these providers will not result in changes to existing files.  The expectation is that after access for yoru initial admin and your proxy nodes is established that all new authorizations are setup via the NiFi UI which will result in updated to these files.  So rename your existing users.xml and authorizations.xml before starting yoru NiFi so new get created.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

 

Translate
apmmahesh
apmmaheshAuthor
Silver ⭐️⭐️
March 23, 2023

Thanks @MattWho  after adding that I am getting new error now in web UI

1.When I try to access node1 in web UI I am getting "Invalid State The Flow Controller is initializing The Data Flow"

2.When I try to access node2 in web UI I am able to access connected/Total number of nodes in the cluster is 1/3

3.When I try to access node3 in web UI I am getting "Invalid State The Flow Controller is initializing The Data Flow"

 

 


node1 nifi-app.log 

2023-03-21 17:53:41,519 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at Node2:11443; will use this address for sending heartbeat messages
2023-03-21 17:53:41,519 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at Node2:11443. Will send Cluster Connection Request to this address
2023-03-21 17:53:41,522 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
node2 nifi-app.log 
WARN [Process Cluster Protocol Request-26] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from node1 due to Received fatal alert: certificate_unknown

WARN [Process Cluster Protocol Request-26] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from node3 due to Received fatal alert: certificate_unknown

node3 nifi-app.log 

2023-03-21 17:53:41,519 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at Node2:11443; will use this address for sending heartbeat messages
2023-03-21 17:53:41,519 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at Node2:11443. Will send Cluster Connection Request to this address
2023-03-21 17:53:41,522 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

node1 verbose output of NiFi keystore # keytool -v -list -keystore keystore.jks

 

Owner: CN=nifi1, OU=NIFI
Issuer: CN=localhost, OU=NIFI
Serial number: 186e9f79f1000000000
Valid from: Thu Mar 16 18:29:24 CST 2023 until: Wed Jun 18 18:29:24 CST 2025
Certificate fingerprints:
MD5: 69:05:14:4A:61:5A:4A:E4:36:2F:8B:C9:E8:C5:D7:EC
SHA1: 1F:98:8A:64:C4:DF:C0:E4:A4:8E:9A:F8:44:9A:DD:24:87:B1:54:A3
SHA256: FA:A8:A0:DC:F3:15:41:28:26:D9:88:70:E9:B8:C9:0F:37:BC:C8:3F:00:E0:A2:D4:6C:55:74:C1:07:90:14:DF
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 8B 2C 24 76 04 D0 83 C8 25 36 12 0D D1 4D A0 57 .,$v....%6...M.W
0010: F9 18 60 BD ..`.
]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:false
PathLen: undefined
]

#3: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
clientAuth
serverAuth
]

#4: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Non_repudiation
Key_Encipherment
Data_Encipherment
Key_Agreement
]

#5: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: nifi1
]

#6: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 33 65 62 1D FB EE 28 01 65 57 F0 12 15 B2 8C 89 3eb...(.eW......
0010: 4B FE A8 A4 K...
]
]

Certificate[2]:
Owner: CN=localhost, OU=NIFI
Issuer: CN=localhost, OU=NIFI
Serial number: 186e9f79e9200000000
Valid from: Thu Mar 16 18:29:24 CST 2023 until: Wed Jun 18 18:29:24 CST 2025
Certificate fingerprints:
MD5: F3:BC:5B:ED:1D:AA:BC:23:17:2E:C1:FB:A5:1F:F6:66
SHA1: AE:BF:55:52:B3:75:24:81:4F:0E:F5:22:62:E9:00:2C:10:87:1A:40
SHA256: 1E:9A:AB:40:E7:C8:82:0B:11:7A:95:3F:73:09:00:29:38:50:59:61:44:BC:3D:0A:42:C2:1F:FC:D0:D7:63:8A
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 8B 2C 24 76 04 D0 83 C8 25 36 12 0D D1 4D A0 57 .,$v....%6...M.W
0010: F9 18 60 BD ..`.
]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:true
PathLen:2147483647
]

#3: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
clientAuth
serverAuth
]

#4: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Non_repudiation
Key_Encipherment
Data_Encipherment
Key_Agreement
Key_CertSign
Crl_Sign
]

#5: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: localhost
]

#6: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 8B 2C 24 76 04 D0 83 C8 25 36 12 0D D1 4D A0 57 .,$v....%6...M.W
0010: F9 18 60 BD ..`.
]
]

 

*******************************************
*******************************************

 

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore keystore.jks -destkeystore keystore.jks -deststoretype pkcs12".

Nifi version: 1.19.1

java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)

I am very new to nifi. Let me know if you need any other information 

 

Translate
MattWho
Titanium ⭐️⭐️⭐️
MattWho
Titanium ⭐️⭐️⭐️
March 23, 2023

@apmmahesh 
Your current issue is unrelated to your original issue.  What you have now is an issue with your mutual TLS exchange between your cluster nodes. NiFi does nothing unusual or out of spec with regards to TLS.

Specifically this line snippet you shared:

 

javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

 

points to a trust issue within the exchange.

So few things about your certificates:
1. Is "nifi1" the actual hostname for this server? (same for nifi2 and nifi3)
2. What is inside the truststore.jks you are using on each of your NiFi nodes?  The complete TLS trustchain for your NiFi certificates must be in this truststore.jks.  Your truststore.jks should contain one too many TrustedCertEntries.  Your keystore.jks must contain only 1 PrivateKeyEntry.

Assuming that all your node's certificates were generated using the NiFi TLS toolkit, they should all be issued/signed by the same certifcate Authority (Issuer: CN=localhost, OU=NIFI).  Their then must be a a trustedCertEntry present in the trustsore.jks used by all your nodes for "CN=localhost, OU=NIFI".

You can use the same keytool -v -list -keystore truststore.jks command to verify the contents of your truststore.jks.

You can also use the following command to obtain the public cert(s) for your CA from any one of your NiFi servers:

openssl s_client -connect <nifi hostname>:<nifi port> -showcerts

each public cert will have this structure:

-----BEGIN CERTIFICATE-----
MIIFYjCCBEqgAw..............b9Sz/yilrbCgj8=
-----END CERTIFICATE----

I cut out portion of string in above, but the certificate includes everything above in it.  You can copy each and save them to "<filename>.crt" files which you can import to your truststore.jks files (all nodes shoudl e using a common truststore.jks) if they are missing.

keytool -import -alias <some unique ca name> -file <filename>.crt -keystore truststore.jks -storepass <truststore.jks password>

 

You'll need to restart your NiFi anytime you modify either the keystore or truststore.  NiFi only reads these files on startup.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

Translate
    apmmahesh
    apmmaheshAuthor
    Silver ⭐️⭐️
    March 27, 2023

    @MattWho Thanks for the support. I have added the all 3 node public cert(s) in single file (nifi.crt).

    After that I run below command in each node.

    keytool -import -alias devnifi -file nifi.crt -keystore truststore.jks -storepass xyz

    For your information
    openssl s_client -connect nifi1.net:9443 -showcerts output of node1

     

    CONNECTED(00000003)
    depth=1 OU = NIFI, CN = localhost
    verify error:num=19:self signed certificate in certificate chain
    140139985950608:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:s3_pkt.c:1493:SSL alert number 42
    140139985950608:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
    ---
    Certificate chain
     0 s:/OU=NIFI/CN=nifi1.net
       i:/OU=NIFI/CN=localhost
    -----BEGIN CERTIFICATE-----
    MIIDjDCCAnSgAwIBAgIKAYbp958QAAAAADANBgkqhkiG9w0BAQsFADAjMQ0wCwYD
    VQQLDAROSUZJMRIwEAYDVQQDDAlsb2NhbGhvc3QwHhcNMjMwMzE2MTAyOTI0WhcN
    MjUwNjE4MTAyOTI0WjA6MQ0wCwYDVQQLDAROSUZJMSkwJwYDVQQDDCB0YW9vZHNo
    ZHBkbjAxLmxpLmx1bWVudHVtaW5jLm5ldDCCASIwDQYJKoZIhvcoAQEBBQADggEP
    ADCCAQoCggEBALXDym1yZ9EiUU30yXp0VmDJvujqcnBnEVqQBVIWXOgBQxbol/Pq
    5J7T3+zd4QPbIggsDeJyZG4kDeWUEbNFv1yiy+zR9atUj8DwRrNPSvgn+6U9DQSp
    1EuEr35IA0VUJNuoPapvCeeOnqHiS7HR92XgrfgOxEW3Qj/phnQAEEz5BYn1RlYn
    k7HklgqOIjkRd7HVZ8JIQEFMUChAzRwdJxa3ty+gMsebGP0K8BWFCeL3DEBlerdj
    agapX91yhTayDT06y59IVQ98rIRs6TDbL/vJpTlbaJqo/UH4EtBUyy8fy9vIHTPF
    4aKgNf23VoN6EqtBSb6y8Qf3wHjoPoQpF1cCAwEAAaOBqjCBpzAdBgNVHQ4EFgQU
    M2ViHfvuKAFlV/ASFbKMiUv+qKQwHwYDVR0jBBgwFoAUiywkdgTQg8glNhIN0U2g
    V/kYYL0wDgYDVR0PAQH/BAQDAgP4MAkGA1UdEwQCMAAwHQYDVR0lBBYwFAYIKwYB
    BQUHAwIGCCsGAQUFBwMBMCsGA1UdEQQkMCKCIHRhb29kc2hkcGRuMDEubGkubHVt
    ZW50dW1pbmMubmV0MA0GCSqGSIb3DQEBCwUAA4IBAQBvhTwotQ2N14q7+mfnFE/g
    RMdQCoMC3Ww+C1/v/ZpMykst4YQAbW0BXPP3wofy604L12G1IhfLiwGGBc+K89Q/
    4HfDsFtQ30g4Ovier3NdwTlLihgHJj36s9ykPox/DSr8oQeKBPFy+++XVLfbtoVz
    GO7HpQMXUwEa2RCpFjDzmAQd2Cc6QjtB1gZ6tT3bjAHHEy963V+LbD1YFSLqNWGw
    r2bkVcRVuCy+jkwFfNn4ZO2U/OYNIfrOiSrektMwbExtgGOKn/drKC3aELeWOmfR
    SeHy5kfxHrFPv2aP+8Dcg61Bj5HQk3B0A9SLRUyJeK7v/XnNypH56oejy6al/Tu5
    -----END CERTIFICATE-----
     1 s:/OU=NIFI/CN=localhost
       i:/OU=NIFI/CN=localhost
    -----BEGIN CERTIFICATE-----
    MIIDYTCCAkmgAwIBAgIKAYbp956SAAAAADANBgkqhkiG9w0BAQsFADAjMQ0wCwYD
    VQQLDAROSUZJMRIwEAYDVQQDDAlsb2NhbGhvc3QwHhcNMjMwMzE2MTAyOTI0WhcN
    MjUwNjE4MTAyOTI0WjAjMQ0wCwYDVQQLDAROSUZJMRIwEAYDVQQDDAlsb2NhbGhv
    c3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/yg1UnKfLvj2LgJ11
    VGeF58CZcR/3DSztOVeBuy+eNHIt9so5D7OWlcMkZzEyxurY4e2gHBNvZowhSK2l
    VO/H4b4rE4FUtmpLu68fXZWgioENF2xQ6xeisVxQuxEvsY6dl7DkyBAvp8zKiuQS
    3momXYZeEDDHXfDx1v0lDSntlQNe4Fb88ROweOqc3CMtyEs/9fhjW/iAFzCmcqC4
    0uvSJyCYJGyMspQ/mRU4SaMqqe0WJ3Je/wwIXxqbdMGksavUCa976OEk0rqPxvf/
    pw6armSwKuZVICt8WGKoQw2rbwEo4jIeU8DdhAcM2GhTdgEJ3WCs+F2gu20meEHw
    a6htAgMBAAGjgZYwgZMwDgYDVR0PAQH/BAQDAgH+MAwGA1UdEwQFMAMBAf8wHQYD
    VR0OBBYEFIssJHYE0IPIJTYSDdFNoFf5GGC9MB8GA1UdIwQYMBaAFIssJHYE0IPI
    JTYSDdFNoFf5GGC9MB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATAUBgNV
    HREEDTALgglsb2NhbGhvc3QwDQYJKoZIhvcNAQELBQADggEBAKpXb5lPlu8iaQI1
    RFGQHjcoGJKRt8C3EHqvEbXeLT1Zck1NAKWsM+YYveX4oLanS5gMPZbcGm7sCaxk
    BkdUP+mMej3+K6fiViexztn2ZRW4u3gV4/dVx/li0Z7Aw1l+Ebm6zSWxujnzJwOP
    BjveQHi7JYj6g4Axb82GUOFyjJraX796sUKiNjeDDBr/mh1nImrGrNgiNzR+a5Ri
    AV4MN9dfQs23fTkPo3YwcbAt1vTphco6owUEwtf7NB+re7fs0rZ+iq3o+d2fGN9W
    +oRJeYeMdesaHaUR6U1SqYyeZ/TNrkRbGuBF79xXBv7azXyZPxNj8Q6iPQ/Pc6H5
    sC6YSao=
    -----END CERTIFICATE-----
    ---
    Server certificate
    subject=/OU=NIFI/CN=nifi1.net
    issuer=/OU=NIFI/CN=localhost
    ---
    Acceptable client certificate CA names
    /OU=NIFI/CN=nifi1.net
    /OU=NIFI/CN=localhost
    Client Certificate Types: RSA sign, DSA sign, ECDSA sign
    Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
    Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
    Peer signing digest: SHA512
    Server Temp Key: ECDH, P-256, 256 bits
    ---
    SSL handshake has read 2363 bytes and written 194 bytes
    ---
    New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
    Server public key is 2048 bit
    Secure Renegotiation IS supported
    Compression: NONE
    Expansion: NONE
    No ALPN negotiated
    SSL-Session:
        Protocol  : TLSv1.2
        Cipher    : ECDHE-RSA-AES256-SHA384
        Session-ID: 6421586CFF9821F1E9851E6E7517D0DB5599F0A20A021033BF64B469610E5D50
        Session-ID-ctx:
        Master-Key: 244C473DE1585F13FFCA6682898903ED4A4287F1A1921D705437DDB7660847F54032543A24D474D14487867FC67BF895
        Key-Arg   : None
        Krb5 Principal: None
        PSK identity: None
        PSK identity hint: None
        Start Time: 1679906924
        Timeout   : 300 (sec)
        Verify return code: 19 (self signed certificate in certificate chain)
    ---

     

    keytool -v -list -keystore truststore.jks output of node1

    Enter keystore password:
    Keystore type: JKS
    Keystore provider: SUN

    Your keystore contains 2 entries

    Alias name: devnifi
    Creation date: Mar 27, 2023
    Entry type: trustedCertEntry

    Owner: CN=nifi1.net, OU=NIFI
    Issuer: CN=localhost, OU=NIFI
    Serial number: 186e9f79f1000000000
    Valid from: Thu Mar 16 18:29:24 CST 2023 until: Wed Jun 18 18:29:24 CST 2025
    Certificate fingerprints:
             SHA1: 1F:98:8A:64:C4:DF:C0:E4:A4:8E:9A:F8:44:9A:DD:24:87:B1:54:A3
             SHA256: FA:A8:A0:DC:F3:15:41:28:26:D9:88:70:E9:B8:C9:0F:37:BC:C8:3F:00:E0:A2:D4:6C:55:74:C1:07:90:14:DF
    Signature algorithm name: SHA256withRSA
    Subject Public Key Algorithm: 2048-bit RSA key
    Version: 3

    Extensions:

    #1: ObjectId: 2.5.29.35 Criticality=false
    AuthorityKeyIdentifier [
    KeyIdentifier [
    0000: 8B 2C 24 76 04 D0 83 C8   25 36 12 0D D1 4D A0 57  .,$v....%6...M.W
    0010: F9 18 60 BD                                        ..`.
    ]
    ]

    #2: ObjectId: 2.5.29.19 Criticality=false
    BasicConstraints:[
      CA:false
      PathLen: undefined
    ]

    #3: ObjectId: 2.5.29.37 Criticality=false
    ExtendedKeyUsages [
      clientAuth
      serverAuth
    ]

    #4: ObjectId: 2.5.29.15 Criticality=true
    KeyUsage [
      DigitalSignature
      Non_repudiation
      Key_Encipherment
      Data_Encipherment
      Key_Agreement
    ]

    #5: ObjectId: 2.5.29.17 Criticality=false
    SubjectAlternativeName [
      DNSName: nifi1.net
    ]

    #6: ObjectId: 2.5.29.14 Criticality=false
    SubjectKeyIdentifier [
    KeyIdentifier [
    0000: 33 65 62 1D FB EE 28 01   65 57 F0 12 15 B2 8C 89  3eb...(.eW......
    0010: 4B FE A8 A4                                        K...
    ]
    ]



    *******************************************
    *******************************************


    Alias name: nifi-cert
    Creation date: Mar 16, 2023
    Entry type: trustedCertEntry

    Owner: CN=localhost, OU=NIFI
    Issuer: CN=localhost, OU=NIFI
    Serial number: 186e9f79e9200000000
    Valid from: Thu Mar 16 18:29:24 CST 2023 until: Wed Jun 18 18:29:24 CST 2025
    Certificate fingerprints:
             SHA1: AE:BF:55:52:B3:75:24:81:4F:0E:F5:22:62:E9:00:2C:10:87:1A:40
             SHA256: 1E:9A:AB:40:E7:C8:82:0B:11:7A:95:3F:73:09:00:29:38:50:59:61:44:BC:3D:0A:42:C2:1F:FC:D0:D7:63:8A
    Signature algorithm name: SHA256withRSA
    Subject Public Key Algorithm: 2048-bit RSA key
    Version: 3

    Extensions:

    #1: ObjectId: 2.5.29.35 Criticality=false
    AuthorityKeyIdentifier [
    KeyIdentifier [
    0000: 8B 2C 24 76 04 D0 83 C8   25 36 12 0D D1 4D A0 57  .,$v....%6...M.W
    0010: F9 18 60 BD                                        ..`.
    ]
    ]

    #2: ObjectId: 2.5.29.19 Criticality=false
    BasicConstraints:[
      CA:true
      PathLen:2147483647
    ]

    #3: ObjectId: 2.5.29.37 Criticality=false
    ExtendedKeyUsages [
      clientAuth
      serverAuth
    ]

    #4: ObjectId: 2.5.29.15 Criticality=true
    KeyUsage [
      DigitalSignature
      Non_repudiation
      Key_Encipherment
      Data_Encipherment
      Key_Agreement
      Key_CertSign
      Crl_Sign
    ]

    #5: ObjectId: 2.5.29.17 Criticality=false
    SubjectAlternativeName [
      DNSName: localhost
    ]

    #6: ObjectId: 2.5.29.14 Criticality=false
    SubjectKeyIdentifier [
    KeyIdentifier [
    0000: 8B 2C 24 76 04 D0 83 C8   25 36 12 0D D1 4D A0 57  .,$v....%6...M.W
    0010: F9 18 60 BD                                        ..`.
    ]
    ]



    *******************************************
    *******************************************

     

    Then I have delete the all 3 node authorizations.xml and users.xml file and restarted the  nifi.sh service and zookeeper service as well.
    same issue in web UI

     

    1.When I try to access node1 in web UI I am getting "Invalid State The Flow Controller is initializing The Data Flow"

    2.When I try to access node2 in web UI I am getting "Invalid State The Flow Controller is initializing The Data Flow"

    3.When I try to access node3 in web UI I am able to access connected/Total number of nodes in the cluster is 1/3

     

    Below are the error log files for all 3 nodes

    node1 nifi-app.log

    2023-03-27 16:25:26,523 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at node3.net:11443; will use this address for sending heartbeat messages
    2023-03-27 16:25:26,523 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at node3.net:11443. Will send Cluster Connection Request to this address
    2023-03-27 16:25:26,527 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors


    node2 nifi-app.log
    2023-03-27 16:23:50,043 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi3.net:11443; will use this address for sending heartbeat messages
    2023-03-27 16:23:50,043 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at nifi3.net:11443. Will send Cluster Connection Request to this address
    2023-03-27 16:23:50,047 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

    node3 nifi-app.log

    6:20:11,079 WARN [Process Cluster Protocol Request-25] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from nifi1.net due to Received fatal alert: certificate_unknown
    2023-03-27 16:20:11,983 INFO [Process Cluster Protocol Request-26] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 2c0d0d2a-b799-4e40-963e-59adb2e6f2d0 (type=HEARTBEAT, length=4315 bytes) from nifi3.net:9443 in 92 millis
    2023-03-27 16:20:11,985 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2023-03-27 16:20:11,888 and sent to nifi3.net:11443 at 2023-03-27 16:20:11,985; determining Cluster Coordinator took 1 millis; DNS lookup for coordinator took 0 millis; connecting to coordinator took 1 millis; sending heartbeat took 51 millis; receiving first byte from response took 42 millis; receiving full response took 43 millis; total time was 97 millis
    2023-03-27 16:20:14,820 WARN [Process Cluster Protocol Request-27] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from nifi2.net due to java.net.SocketException: Broken pipe (Write failed)
    java.net.SocketException: Broken pipe (Write failed)
    at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
    at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
    at java.base/sun.security.ssl.SSLSocketOutputRecord.flush(SSLSocketOutputRecord.java:271)
    at java.base/sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:89)
    at java.base/sun.security.ssl.Finished$T13FinishedProducer.onProduceFinished(Finished.java:773)
    at java.base/sun.security.ssl.Finished$T13FinishedProducer.produce(Finished.java:661)
    at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)
    at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.goServerHello(ClientHello.java:1223)
    at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(ClientHello.java:1159)
    at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:841)
    at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:802)
    at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
    at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183)
    at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
    at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1507)
    at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1417)
    at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
    at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:922)
    at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1013)
    at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:973)
    at org.apache.nifi.stream.io.ByteCountingInputStream.read(ByteCountingInputStream.java:42)
    at java.base/java.io.FilterInputStream.read(FilterInputStream.java:83)
    at org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$2.unmarshal(JaxbProtocolContext.java:113)
    at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:150)
    at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:131)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
    2023-03-27 16:20:15,996 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 83047 nanos
    2023-03-27 16:20:16,089 WARN [Process Cluster Protocol Request-28] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from nifi1.net due to java.net.SocketException: Broken pipe (Write failed)
    java.net.SocketException: Broken pipe (Write failed)
    at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
    at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
    at java.base/sun.security.ssl.SSLSocketOutputRecord.flush(SSLSocketOutputRecord.java:271)
    at java.base/sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:89)
    at java.base/sun.security.ssl.CertificateRequest$T12CertificateRequestProducer.produce(CertificateRequest.java:636)
    at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)
    at java.base/sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1091)
    at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:843)
    at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:802)
    at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
    at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183)
    at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
    at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1507)
    at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1417)
    at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
    at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:922)
    at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1013)
    at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:973)
    at org.apache.nifi.stream.io.ByteCountingInputStream.read(ByteCountingInputStream.java:42)
    at java.base/java.io.FilterInputStream.read(FilterInputStream.java:83)
    at org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$2.unmarshal(JaxbProtocolContext.java:113)
    at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:150)
    at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:131)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

     

    Regards

    Mahesh

     

     

    Translate
    MattWho
    Titanium ⭐️⭐️⭐️
    MattWho
    Titanium ⭐️⭐️⭐️
    March 27, 2023

    @apmmahesh 

    Make sure that the nifi.properties file on all nodes is configured the same.
    Make sure that the "nifi.cluster.protocol.is.secure" property is set to true on all the nodes.

    Matt

    Translate
    Terms & ConditionsAccessibility statement