Cannot access the NiFi Registry from NiFi and view the buckets

[#1]

Hello dear community,

I haven’t been able to find a solution for the following setup:
I have Apache NiFi 2.5.0 and NiFi Registry 2.5.0 installed on the same EC2 instance (Linux 2023). Both NiFi and NiFi Registry are configured with SSL and served through an Application Load Balancer (ALB).
External access is only available via the ALB: https://alb...com.
Internally, the services can be reached at:

NiFi: https://nifi_ec2server.com:8443/nifi

NiFi Registry: https://nifi_ec2server.com:18443/nifi-registry

Authentication is handled via Active Directory (LDAP).
Accessing NiFi through https://alb.../nifi works fine — I can log in as an LDAP user. The same applies to NiFi Registry under https://alb.../nifi-registry.

The problem:
As an LDAP user, I cannot access the NiFi Registry from within NiFi, and no buckets are displayed.
The error message is attached below.
[NiFi Registry Web Server-50] o.a.n.r.w.s.a.IdentityFilter Attempting to extract user credentials using X509IdentityProvider
[NiFi Registry Web Server-50] o.a.n.r.w.s.a.x.X509CertificateExtractor No client certificate found in request.
[NiFi Registry Web Server-50] o.a.n.r.w.s.a.IdentityFilter Attempting to extract user credentials using JwtIdentityProvider
[NiFi Registry Web Server-50] o.a.n.r.s.a.BearerAuthIdentityProvider HTTP Bearer Auth credentials not present. Not attempting to extract credentials for authentication.
[NiFi Registry Web Server-50] o.a.n.r.w.s.a.AnonymousIdentityFilter Set SecurityContextHolder to anonymous SecurityContext
[NiFi Registry Web Server-50] o.a.n.r.w.s.a.ResourceAuthorizationFilter Request filter authorization check is not required for this HTTP Method on this resource. Allowing request to proceed. An additional authorization check might be performed downstream of this filter.
[NiFi Registry Web Server-50] o.a.n.r.s.a.r.ProxyChainAuthorizable Requested resource is /buckets
[NiFi Registry Web Server-50] o.a.n.r.s.a.r.ProxyChainAuthorizable Requested resource is /buckets/edfb5073-e6ee-4d86-add6-5d0e9e803f10
[NiFi Registry Web Server-50] o.a.n.r.s.a.r.PublicCheckingAuthorizable Requested resource is /buckets/edfb5073-e6ee-4d86-add6-5d0e9e803f10
[NiFi Registry Web Server-50] o.a.n.r.s.a.r.PublicCheckingAuthorizable Delegating to inheriting authorizable for /buckets/edfb5073-e6ee-4d86-add6-5d0e9e803f10

NiFi Registry is configured as follows:

<!-- Authorizer.xml-->
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<authorizers>

<userGroupProvider>
<identifier>ldap-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider</class>

<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">CN=LDAPUSER1234,OU=Service Users,OU=User Accounts,DC=corp1,DC=ad1,DC=xyz,DC=net</property>
<property name="Manager Password">secret</property>

<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldaps://ldapserver.com:636</property>

<property name="User Search Base">OU=User Accounts,DC=corp1,DC=ad1,DC=xyz,DC=net</property>
<property name="User Search Scope">SUBTREE</property>
<property name="User Object Class">user</property>
<property name="User Search Filter">(&amp;(objectClass=user)(sAMAccountName=*))</property>
<property name="User From Name Filter">(&amp;(objectClass=user)(sAMAccountName={0}))</property>

<property name="Group Search Base">OU=Groups,OU=User Accounts,DC=corp1,DC=ad1,DC=xyz,DC=net</property>
<property name="Group Object Class">group</property>
<property name="Group Search Scope">SUBTREE</property>
<property name="Group Member Attribute">member</property>
<property name="Group Name Attribute">cn</property>

<property name="Group Search Filter">(&amp;(objectClass=group)(member={0}))</property>
<property name="User Group Name Attribute - Referenced User Attribute">distinguishedName</property>
<property name="Sync Interval">30 mins</property>

<property name="Group Membership Searching">limited</property>
<property name="Max Group Membership Search Level">1</property>

<property name="GUID Attribute">objectGUID</property>
<property name="Authentication Expiration">12 hours</property>

</userGroupProvider>

<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
<property name="User Group Provider">ldap-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Users File">./conf/users.xml</property>
<property name="Initial Admin Identity">CN=ldap_employee,OU=User Accounts,DC=corp1,DC=ad1,DC=xyz,DC=net</property>
<property name="Initial Admin Read Only">false</property>

</accessPolicyProvider>

<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers>

######################################################################-----------------------------------------------------#####################################################################

<!-- identity-providers.xml -->

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<identityProviders>
<provider>
<identifier>ldap-provider</identifier>
<class>org.apache.nifi.registry.security.ldap.LdapIdentityProvider</class>

<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">CN=LDAPUSER1234,OU=Service Users,OU=User Accounts,DC=corp1,DC=ad1,DC=xyz,DC=net</property>
<property name="Manager Password">secret</property>

<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldaps://ldapserver.com:636</property>

<!-- Sucht in allen User-OU-Bereichen -->
<property name="User Search Base">OU=User Accounts,DC=corp1,DC=ad1,DC=xyz,DC=net</property>
<property name="User Search Scope">SUBTREE</property>
<property name="User Identity Attribute">sAMAccountName</property>
<property name="User Object Class">user</property>
<property name="User Search Filter">(&amp;(objectClass=user)(sAMAccountName={0}))</property>
<property name="Identity Strategy">USE_DN</property>
<property name="Authentication Expiration">12 hours</property>
</provider>
</identityProviders>

######################################################################-----------------------------------------------------#####################################################################

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# web properties #
nifi.registry.web.war.directory=./lib
nifi.registry.web.https.host=nifi_ec2server.com
nifi.registry.web.https.port=18443
nifi.registry.web.https.network.interface.default=
nifi.registry.web.https.application.protocols=h2 http/1.1
nifi.registry.web.jetty.working.directory=./work/jetty
nifi.registry.web.jetty.threads=200
nifi.registry.web.should.send.server.version=true

# External URL via ALB (important for redirects)
nifi.registry.web.proxy.host=nifi_alb_server.com

# security properties #
nifi.registry.security.keystore=./conf/keystore_new.jks
nifi.registry.security.keystoreType=JKS
nifi.registry.security.keystorePasswd=secret
nifi.registry.security.keyPasswd=secret
nifi.registry.security.truststore=./conf/truststore.jks
nifi.registry.security.truststoreType=JKS
nifi.registry.security.truststorePasswd=secret
nifi.registry.security.authorizers.configuration.file=./conf/authorizers.xml
nifi.registry.security.authorizer=managed-authorizer
nifi.registry.security.identity.providers.configuration.file=./conf/identity-providers.xml
nifi.registry.security.identity.provider=ldap-provider
nifi.registry.security.user.login.identity.provider=ldap-provider
nifi.registry.security.user.group.provider=ldap-user-group-provider
nifi.registry.security.authorization.provider=file-access-policy-provider

nifi.registry.security.needClientAuth=false

# --------------------------------------------------
# LDAP Configuration
# --------------------------------------------------
ldap-provider.url=ldaps://ldapserver.com:636
ldap-provider.user.search.base=OU=User Accounts,DC=corp1,DC=ad1,DC=xyz,DC=net
ldap-provider.user.search.filter=(&amp;(objectClass=user)(sAMAccountName=*))
ldap-provider.group.search.base=OU=User Accounts,DC=corp1,DC=ad1,DC=xyz,DC=net
ldap-provider.group.search.filter=(&amp;(objectClass=group)(member={0}))
ldap-provider.bind.user=CN=LDAPUSER1234,OU=Service Users,OU=User Accounts,DC=corp1,DC=ad1,DC=xyz,DC=net
ldap-provider.bind.password=secret

# nifi.registry.security.user.login.identity.provider=ldap-provider

# providers properties #
nifi.registry.providers.configuration.file=./conf/providers.xml

# registry alias properties #
nifi.registry.registry.alias.configuration.file=./conf/registry-aliases.xml

# extensions working dir #
nifi.registry.extensions.working.directory=./work/extensions

# legacy database properties, used to migrate data from original DB to new DB below
# NOTE: Users upgrading from 0.1.0 should leave these populated, but new installs after 0.1.0 should leave these empty
nifi.registry.db.directory=
nifi.registry.db.url.append=

# database properties
nifi.registry.db.url=jdbc:h2:./database/nifi-registry-primary;AUTOCOMMIT=OFF;DB_CLOSE_ON_EXIT=FALSE;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
nifi.registry.db.driver.class=org.h2.Driver
nifi.registry.db.driver.directory=
nifi.registry.db.username=nifireg
nifi.registry.db.password=nifireg
nifi.registry.db.maxConnections=5
nifi.registry.db.sql.debug=false

# extension directories #
# Each property beginning with "nifi.registry.extension.dir." will be treated as location for an extension,
# and a class loader will be created for each location, with the system class loader as the parent
#
#nifi.registry.extension.dir.1=/path/to/extension1
#nifi.registry.extension.dir.2=/path/to/extension2

nifi.registry.extension.dir.aws=./ext/aws/lib

# Identity Mapping Properties #
# These properties allow normalizing user identities such that identities coming from different identity providers
# (certificates, LDAP, Kerberos) can be treated the same internally in NiFi. The following example demonstrates normalizing
# DNs from certificates and principals from Kerberos into a common identity string:
#
# nifi.registry.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), O=(.*?), L=(.*?), ST=(.*?), C=(.*?)$
# nifi.registry.security.identity.mapping.value.dn=$1@$2
# nifi.registry.security.identity.mapping.transform.dn=NONE

nifi.registry.security.identity.mapping.pattern.dn=^CN=(.*?),.*$
nifi.registry.security.identity.mapping.value.dn=$1

# nifi.registry.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
# nifi.registry.security.identity.mapping.value.kerb=$1@$2
# nifi.registry.security.identity.mapping.transform.kerb=UPPER

# Group Mapping Properties #
# These properties allow normalizing group names coming from external sources like LDAP. The following example
# lowercases any group name.
#
# nifi.registry.security.group.mapping.pattern.anygroup=^(.*)$
# nifi.registry.security.group.mapping.value.anygroup=$1
# nifi.registry.security.group.mapping.transform.anygroup=LOWER

# kerberos properties #
# nifi.registry.kerberos.krb5.file=
# nifi.registry.kerberos.spnego.principal=
# nifi.registry.kerberos.spnego.keytab.location=
# nifi.registry.kerberos.spnego.authentication.expiration=12 hours

# OIDC #
# nifi.registry.security.user.oidc.discovery.url=
# nifi.registry.security.user.oidc.connect.timeout=
# nifi.registry.security.user.oidc.read.timeout=
# nifi.registry.security.user.oidc.client.id=
# nifi.registry.security.user.oidc.client.secret=
# nifi.registry.security.user.oidc.preferred.jwsalgorithm=
# nifi.registry.security.user.oidc.additional.scopes=${nifi.registry.security.user.oidc.additional.scopes}
# nifi.registry.security.user.oidc.claim.identifying.user=${nifi.registry.security.user.oidc.claim.identifying.user}
# nifi.registry.security.user.oidc.claim.groups=groups

# revision management #
# This feature should remain disabled until a future NiFi release that supports the revision API changes
nifi.registry.revisions.enabled=false

Thank you & best regards,
casaui

[#2]

@casaui 

You have multiple issues with your setup.

Issue 1:
Problem:  Missing ClientAuth certificate in MutualTLS exchange between NiFi's NiFiRegistryFlowRegistryClient and your NiFi-Registry.

[NiFi Registry Web Server-50] o.a.n.r.w.s.a.x.X509CertificateExtractor No client certificate found in request.

Solution:
The NiFiRegistryFlowRegistryClient will use the keystore and truststore defined in the nifi.properties file if a SSL Context service has not been configured in the registry client directly.  For a mutualTLS exchange to be successful, the client side must present a clientAuth certificate that is trusted by the truststore being used on the server side (NiFi-Registry) of the TLS connection.  Likewise, the serverAuth certificate presented by the server (NiFi-Registry) must be trusted by the truststore the client (NiFiFlowRegistryClient) is configured to use.   I have no idea why you have an ALB setup for NiFi-registry since there is just one NiFi-Registry.  The NiFiRegistryFlowRegistryClient should be configured with the direct NiFi-Registry URL: https://nifi_ec2server.com:18443/nifi-registry
Without a clientAuth certificate presented in the mutualTLS exchange your Registry client is going to be anonymous.

Issue 2: Authorization
Problem:  Every action is authenticated and authorized when it comes to interacting with a secured NiFi or NiFi-Registry. When the Registry Client attempts to establish a connection to the NiFi-registry, it uses the DN from clientAuth certificate from the configured keystore (provided server can trust it) as the client/user identity.   From your shared authorizers.xml I can see you are using the managed authorizer --> file-access-policy-provider --> ldap-user-group-provider.   So unless your ldap is returning the DN for your clientAuth certificate, you won't be able to setup the necessary authorization needed for the client.
Solution: The authorizers.xml could be setup instead to use managed-authorizer --> file-access-policy-provider (modified to use composite-configurable-user-group-provider instead of ldap-user-group-provider) --> composite-configurable-user-group-provider (configured to use configurable provider = file-user-group-provider and provider ldap-user-group-provider).    Both the file-user-group-provider and ldap-user-group-provider configured in the composite-configurable-user-group-provider need to also be configured.  The file-user-group-provider will allow you manually added client/user identities not provided via your ldap-user-group-provider to which you can set authorization policies needed.  The clientAuth certificate DN (may be unique for each NiFi node in a NiFi cluster) would need to be authorized for "Can proxy user requests (read, write, delete)" and "Can Manage Buckets (read)".

Issue 3: Authorizers - ldap-user-group-provider use of {0}.
Problem:  I suspect you are not getting any users returned by this provider because your user search filter is using "sAMAccountName={0}" and group search filter is using "member={0}".  "{0}" is going to treated as a literal string.  It only has special meaning in the ldap-provider.  
Solution:  Stop using {0} in the authorizers.xml file ldap-user-group-provider.  This provider executes every 30 minutes to sync users and groups from ldap.  It doe snot take any outside input.  The ldap-provider does however.  The username entered at login is inserted in place of the {0} in the ldap-provider to be used to verify the username and password provided at login.

Issue 4: Authorizers and login-provider - Auth strategy mismatch
Problem:  You configured auth strategy as Simple; however, configured a ldaps url:

<property name="Authentication Strategy">SIMPLE</property>
...
<property name="Url">ldaps://ldapserver.com:636</property>

Solution: This provider is not going to work configured like this.  Auth strategy should be "LDAPS" when connecting to "ldaps://..." or your Url should be unsecured "ldap://..." if you still want to use "SIMPLE"

------
Your user identity as displayed (case sensitive) in the upper right corner of the NiFi UI, must also exist as a user in NiFi-Registry and be properly authorized on the bucket(s) that user can use for version control.  Keep in mind that the NiFiRegistryFlowRegistryClient will proxy read/write request on behalf of your NiFi user.  So the authorizations I stated early for the registry client and the authorizations for the NiFi user must both exist in NiFi-Registry.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt