Hello Community,
I'm trying to set-up a secured 3 node NiFi cluster with self-signed certs, generated via Cert-Manager. Nifi is deployed via a Helm chart into AKS.
The issue is, as soon as I set the number of nodes to 3 I get this error. With 1 node, I did not have this issue.
So, all 3 nodes has their own Keystore and a shared Truststore. All 3 nodes' TLS cert is stored in the shared truststore.
I have my authorizers.xml set up:
<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./auth-conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<property name="Initial User Identity 1">initial.admin@mail.com</property>
<property name="Initial User Identity 3">CN=nifi-0</property>
<property name="Initial User Identity 4">CN=nifi-1</property>
<property name="Initial User Identity 5">CN=nifi-2</property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">./auth-conf/authorizations.xml</property>
<property name="Initial Admin Identity">initial.admin@mail.com</property>
<property name="Node Identity 1">CN=nifi-0</property>
<property name="Node Identity 2">CN=nifi-1</property>
<property name="Node Identity 3">CN=nifi-2</property>
</accessPolicyProvider>
The TLS certs for each node contains the same names: CN=nifi-0, etc.
I've made sure that I have deleted the authorizations.xml and users.xml on my nodes so it will be generated again up to date.
All my nodes has the /proxy policy on all my nodes:
<authorizations>
<policies>
<policy identifier="f99bccd1-a30e-3e4a-98a2-dbc708edc67f" resource="/flow" action="R">
<user identifier="08106046-f063-3605-9480-9afc3f4726d4"/>
</policy>
<policy identifier="17b6f3de-feaf-3539-b938-ee287e625cf3" resource="/data/process-groups/9701352c-a1c0-4cfc-a6bb-92f76382e007" action="R">
<user identifier="08106046-f063-3605-9480-9afc3f4726d4"/>
<user identifier="5d994b80-c1b1-3c50-aa9f-488c1e549c53"/>
<user identifier="8fa66807-0e9c-3972-b28e-ed8542fec384"/>
<user identifier="a3a07ec3-6507-3e19-bc6a-c47e0deca8c6"/>
</policy>
<policy identifier="6bf50b6c-d24b-38a4-9c6e-1b9bd888b6ee" resource="/data/process-groups/9701352c-a1c0-4cfc-a6bb-92f76382e007" action="W">
<user identifier="08106046-f063-3605-9480-9afc3f4726d4"/>
<user identifier="5d994b80-c1b1-3c50-aa9f-488c1e549c53"/>
<user identifier="8fa66807-0e9c-3972-b28e-ed8542fec384"/>
<user identifier="a3a07ec3-6507-3e19-bc6a-c47e0deca8c6"/>
</policy>
<policy identifier="23d164f5-1bef-3c18-86ee-e1b88fd2845f" resource="/process-groups/9701352c-a1c0-4cfc-a6bb-92f76382e007" action="R">
<user identifier="08106046-f063-3605-9480-9afc3f4726d4"/>
</policy>
<policy identifier="d4fd2dc1-5c49-370d-bb81-107b6de0bdcc" resource="/process-groups/9701352c-a1c0-4cfc-a6bb-92f76382e007" action="W">
<user identifier="08106046-f063-3605-9480-9afc3f4726d4"/>
</policy>
<policy identifier="b8775bd4-704a-34c6-987b-84f2daf7a515" resource="/restricted-components" action="W">
<user identifier="08106046-f063-3605-9480-9afc3f4726d4"/>
</policy>
<policy identifier="627410be-1717-35b4-a06f-e9362b89e0b7" resource="/tenants" action="R">
<user identifier="08106046-f063-3605-9480-9afc3f4726d4"/>
</policy>
<policy identifier="15e4e0bd-cb28-34fd-8587-f8d15162cba5" resource="/tenants" action="W">
<user identifier="08106046-f063-3605-9480-9afc3f4726d4"/>
</policy>
<policy identifier="ff96062a-fa99-36dc-9942-0f6442ae7212" resource="/policies" action="R">
<user identifier="08106046-f063-3605-9480-9afc3f4726d4"/>
</policy>
<policy identifier="ad99ea98-3af6-3561-ae27-5bf09e1d969d" resource="/policies" action="W">
<user identifier="08106046-f063-3605-9480-9afc3f4726d4"/>
</policy>
<policy identifier="2e1015cb-0fed-3005-8e0d-722311f21a03" resource="/controller" action="R">
<user identifier="08106046-f063-3605-9480-9afc3f4726d4"/>
</policy>
<policy identifier="c6322e6c-4cc1-3bcc-91b3-2ed2111674cf" resource="/controller" action="W">
<user identifier="08106046-f063-3605-9480-9afc3f4726d4"/>
</policy>
<policy identifier="287edf48-da72-359b-8f61-da5d4c45a270" resource="/proxy" action="W">
<user identifier="5d994b80-c1b1-3c50-aa9f-488c1e549c53"/>
<user identifier="8fa66807-0e9c-3972-b28e-ed8542fec384"/>
<user identifier="a3a07ec3-6507-3e19-bc6a-c47e0deca8c6"/>
</policy>
</policies>
</authorizations>
And my users.xml:
<tenants>
<groups/>
<users>
<user identifier="08106046-f063-3605-9480-9afc3f4726d4" identity="initial.admin@mail.com"/>
<user identifier="5d994b80-c1b1-3c50-aa9f-488c1e549c53" identity="CN=nifi-0"/>
<user identifier="8fa66807-0e9c-3972-b28e-ed8542fec384" identity="CN=nifi-1"/>
<user identifier="a3a07ec3-6507-3e19-bc6a-c47e0deca8c6" identity="CN=nifi-2"/>
</users>
</tenants>
User-log error:
2021-08-01 18:35:27,868 INFO [NiFi Web Server-19] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<initial.admin@mail.com><CN=nifi-1>) GET https://nifi-2.nifi-headless.test-nifi.svc.cluster.local:9443/nifi-api/flow/current-user (source ip: x.x.x.x)
2021-08-01 18:35:27,869 WARN [NiFi Web Server-19] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy CN=nifi-1
When I try to log-in on the UI:
Insufficient permissions: Untrusted proxy CN=nifi-1 or nifi-2 or nifi-0
Only other issue which I couldnt figure out yet: Failed to send message to Cluster Coordinator due to: java.net.UnknownHostException: nifi-1.nifi-headless.test-nifi.svc.cluster.local
My pods can resolve each-other and the zookeeper pods aswell.
What can be the issue here?
