Nifi nodes can't access Nifi Registry buckets

[#1]

Both Nifi and Nifi Registry services are configured to use LDAP, the user authentication in both of them are working just fine. The communication between Nifi and Registry is made by calling IP and Port and I use the same addresses to access their UI and login with my LDAP user.

But when I try to start version control in one of the Nifi access groups it can connects to the Nifi Registry but returns "No available buckets" even with the buckets already been created in Nifi Registry.

This is the Registry logs when I use the "Start version control" option inside Nifi and it can't find the buckets:

2024-10-17 17:39:54,289 DEBUG [NiFi Registry Web Server-15] o.a.n.r.w.s.a.IdentityFilter Attempting to extract user credentials using X509IdentityProvider
2024-10-17 17:39:54,290 DEBUG [NiFi Registry Web Server-15] o.a.n.r.w.s.a.x.X509CertificateExtractor No client certificate found in request.
2024-10-17 17:39:54,290 DEBUG [NiFi Registry Web Server-15] o.a.n.r.w.s.a.IdentityFilter Attempting to extract user credentials using JwtIdentityProvider
2024-10-17 17:39:54,290 DEBUG [NiFi Registry Web Server-15] o.a.n.r.s.a.BearerAuthIdentityProvider HTTP Bearer Auth credentials not present. Not attempting to extract credentials for authentication.
2024-10-17 17:39:54,290 DEBUG [NiFi Registry Web Server-15] o.a.n.r.w.s.a.AnonymousIdentityFilter Set SecurityContextHolder to anonymous SecurityContext
2024-10-17 17:39:54,293 DEBUG [NiFi Registry Web Server-15] o.a.n.r.w.s.a.ResourceAuthorizationFilter Request filter authorization check is not required for this HTTP Method on this resource. Allowing request to proceed. An additional authorization check might be performed downstream of this filter.
2024-10-17 17:39:54,685 DEBUG [NiFi Registry Web Server-15] o.a.n.r.s.a.r.ProxyChainAuthorizable Requested resource is /buckets
2024-10-17 17:39:54,788 DEBUG [NiFi Registry Web Server-15] o.a.n.r.s.a.r.ProxyChainAuthorizable Requested resource is /buckets/bd21e50a-67b3-4d1e-9336-5b9324dd8440
2024-10-17 17:39:54,825 DEBUG [NiFi Registry Web Server-15] o.a.n.r.s.a.r.PublicCheckingAuthorizable Requested resource is /buckets/bd21e50a-67b3-4d1e-9336-5b9324dd8440
2024-10-17 17:39:54,860 DEBUG [NiFi Registry Web Server-15] o.a.n.r.s.a.r.PublicCheckingAuthorizable Delegating to inheriting authorizable for /buckets/bd21e50a-67b3-4d1e-9336-5b9324dd8440
2024-10-17 17:39:54,920 DEBUG [NiFi Registry Web Server-15] o.a.n.r.s.a.r.ProxyChainAuthorizable Requested resource is /buckets/f7c12415-4d95-4b5a-a213-1f12d615cc8f
2024-10-17 17:39:55,053 DEBUG [NiFi Registry Web Server-15] o.a.n.r.s.a.r.PublicCheckingAuthorizable Requested resource is /buckets/f7c12415-4d95-4b5a-a213-1f12d615cc8f
2024-10-17 17:39:55,055 DEBUG [NiFi Registry Web Server-15] o.a.n.r.s.a.r.PublicCheckingAuthorizable Delegating to inheriting authorizable for /buckets/f7c12415-4d95-4b5a-a213-1f12d615cc8f

 Nifi has 3 nodes configured in it and I created one user for each one inside Registry and granted all possible permissions to them:

Even ChatGPT wasn't able to resolve this issue... I don't know what I must do to make this works.

[#2]

@HenriqueAX Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @MattWho @SAMSAL  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3]

@HenriqueAX 

The NiFi nodes themselves require a few authorizations within NiFi registry in order to start version control of a process group in NiFi.

The NiFi nodes must be authorized in NiFi-Registry with:
1. "Can proxy user requests" - Read, Write, and Delete
2. "Can manage buckets" - Read

Within NiFi you have setup your "Registry Client" that looks like this:

For secure (HTTPS) NiFi-Registry URL, when an SSL Context Service is not defined, the default keystore and truststore configured in the NiFi.properties file is used to authenticate with NiFi-Registry through a mutual TLS exchange. 

1. Did you setup an SSL Context Service?
2. If so, what keystore and truststore did you configure in the service?
3. Does the truststore used in the nifi-registry.properties file contain the necessary public certificates to include the complete trustchain for your NiFi keystore ClientAuth certificates?

From the log snippet shared, it looks like the mutual TLS exchange is not resulting in a trusted clientAuth certificate being passed in the exchange (commonly a trust chain issue). So NiFi-Registry then checks for token in connection which would not exist in a NiFi to NiFi-Registry connection.  So what ends up happening is the connection is established as anonymous (just like you as a user sees when accessing NiFi before clicking login button).

2024-10-17 17:39:54,289 DEBUG [NiFi Registry Web Server-15] o.a.n.r.w.s.a.IdentityFilter Attempting to extract user credentials using X509IdentityProvider
2024-10-17 17:39:54,290 DEBUG [NiFi Registry Web Server-15] o.a.n.r.w.s.a.x.X509CertificateExtractor No client certificate found in request.
2024-10-17 17:39:54,290 DEBUG [NiFi Registry Web Server-15] o.a.n.r.w.s.a.IdentityFilter Attempting to extract user credentials using JwtIdentityProvider
2024-10-17 17:39:54,290 DEBUG [NiFi Registry Web Server-15] o.a.n.r.s.a.BearerAuthIdentityProvider HTTP Bearer Auth credentials not present. Not attempting to extract credentials for authentication.
2024-10-17 17:39:54,290 DEBUG [NiFi Registry Web Server-15] o.a.n.r.w.s.a.AnonymousIdentityFilter Set SecurityContextHolder to anonymous SecurityContext

So you most likely need to resolve your trust between NiFi and NiFi-Registry.  
Typically both NiFi and NiFi-Registry would use the same truststore containing all intermediate and root certificate authorities in the trust chain for the keystore certificates used on both services.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#4]

@MattWho I exported the keystore content of all 3 Nifi nodes and imported them into the Nifi Registry truststore, and also did the other way around, exported the Nifi Registry keystore to the Nifi truststore, but it keeps returning the same thing in the logs.

When I made the first configs of Registry, I created self signed keystore and truststore and it was probably done the same way for Nifi. I don't know if this could be the cause of this problem...

[#5 · Accepted solution]

@HenriqueAX 

The NiFi keystore contains a private key certificate.
The NiFi Truststore contains trusted cert entries (public certificates).

You should combine all the truststores to make one truststore containing all the public certificates and use that same truststore on all the NiFi nodes and NiFi-Registry host.

It may also help to understand what is happening by looking at the output from openssl:

openssl s_client -connect <nifi hostname>:<nifi port> -showcerts

openssl s_client -connect <nifi-registry hostname>:<nifi-registry port> -showcerts

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#6]

@HenriqueAX Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. 

Thanks.