Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NIFI: "No available buckets" for saving flow version to nifi registry

avatar
New Contributor

Hello all,

 

NoBuckets.PNG

Here is our setup:

-set up secure nifi
-set up secure registry
-both on the same machine
-created user in registry with CN=[hostname], OU=NiFi with read right on buckets and proxy user requests enabled
-created Registry client in the nifi instance with https://[hostname]:[port]
-when trying to start version control on a process group, buckets are loading for a split second, then showing -> "no available buckets"
-activated logging on DEBUG level, can see the proxy request, but no authorization
-when listening to the registry port with netstat, we can see registry listening on that port
-when trying to retrieve buckets through nifi, connection is established, but no buckets can be seen

 

 

Additional information:
We have created buckets, adding both the admin user and the user which is used for the connection in registry to their policies. 
We encounter no error when trying to retrieve the buckets, neither in the UI nor anywhere in the logs (with highest logging level activated).
We have seen several set up videos and many open threads but the suggested solutions are not solving our problem.
We suspect a permission problem, but are unable to detect the root of the problem
if necessary logs can be provided.

7 REPLIES 7

avatar
Super Mentor

@JelenaS 

You are correct that this sounds like an authorization issue.

I recommend tailing the nifi-registry-app.log  and then perform the action of trying to version control a Process Group within NiFI's UI.

How are you handling user authorization in your NiFi and NiFi-Registry?
- File based authorization (users.xml and authorizations.xml)

What identity.mapping patterns have you configured in your NiFi and NiFi-Registry?
How are you authenticating your user that access both NiFi and NiFi-Registry?

The only buckets that would be returned are those buckets for which the Authenticated user in NiFi has access to in NiFi-Registry.
Screen Shot 2020-12-23 at 3.43.29 PM.png

Keep in mind that the user/client strings in NiFi that are passed to NiFi-Registry must match exactly.

Nodes will pass their full DN when they proxy the request on behalf of the authorized user.  The user string will passed as is.   That means identity mapping patterns will be applied on NiFi-Registry side against those NIFi DNs.  Resulting mapped value must match the client string add as a user in NiFi-Registry.

The passed user string must match exactly (case sensitive) or it is treated as a different user.

 

Hope this helps,

Matt

 

avatar
New Contributor

Thank you @MattWho 

 

Our whole setup is very basic so far. We are working with the admin user and authenticate with the generated client certificates (nifi toolkit). We have no additional user authorization in place yet. Therefore also no identity mapping patterns. 

Also used the exact domain name as user CN.

 

When trying to perform version control, the nifi registry log gives the requested resource followed by a repetition of these messages: 

 

IdentityFilter Attempting to extract user credentials using X509IdentityProvider

IdentityFilter Adding credentials claim to SecurityContext to be authenticated. Credentials extracted by X509IdentityProvider: AuthenticationRequest{username='CN=nifi_admin, OU=NiFi', credentials=[PROTECTED], details=org.apache.nifi.registry.web.security.authentication.x509.X509AuthenticationRequestDetails@3938d0ef}

IdentityFilter Credentials already extracted for [org.apache.nifi.registry.web.security.authentication.AuthenticationRequestToken$1@138d2a4], skipping credentials extraction filter using JwtIdentityProvider

AnonymousIdentityFilter SecurityContextHolder not populated with anonymous token, as it already contained: 'AuthenticationRequest{username='CN=nifi_admin, OU=NiFi', credentials=[PROTECTED], details=org.apache.nifi.registry.web.security.authentication.x509.X509AuthenticationRequestDetails@1a175871}'

ResourceAuthorizationFilter Request filter authorization check is not required for this HTTP Method on this resource. Allowing request to proceed. An additional authorization check might be performed downstream of this filter.

ProxyChainAuthorizable Requested resource is /tenants

 

 

This is the only remarkable message but I don't see any obvious error.

avatar
Super Mentor

@JelenaS 

 

You would need to share some screenshots of the policies/permissions you have set on the bucket(s) you have created in your NiFi-Registry. 
- Go to "Settings" (wrench icon upper right corner within NiFi-Registry
- under "BUCKETS" click pencil icon for bucket you expect your user to see
- Your NiFi user which is logged in to NiFi should have write,delete,read on the bucket.

Would also be helpful to what "Special Privileges" you have set for each of your NiFi nodes inside NiFi-Registry as well.
- Go to "Settings" (wrench icon upper right corner within NiFi-Registry
- under "USERS" click pencil icon for each of your NiFi nodes
- Each of your NiFi nodes (case sensitive) should have "Can proxy user requests" and read on "Can manage buckets" checked.

avatar
New Contributor

@MattWho 

We tried two different bucket settings: one publicly visible, the other not

Bucket2.PNGBucket1.PNG

We are logged into NiFi and to NiFi Registry with the NiFi Admin user. Could this cause a problem? 

 

Admin_User.PNGhost_user.PNG

The NiFI Controller Settings look like this:

Registry_client.PNG

 

 

  

avatar
New Contributor

I know this post is rather old and probably no longer of interest for the original user, but as I faced the same or similar issue today and as it took me round about two hours to finally have a docker container with a running secured nifi-registry setup, I wanted to share what I was missing:

 

Besides the admin user for the registry, one must create at least *two* additional users in the registry.

One user must match the running secured NiFi certificate issuer exactly (capital/lowercase and spaces) like: `CN=nifimain, OU=NIFI`. This one needs the general "Can proxy user requests" rights. The other user created must match exactly the user you use to login into your NiFi web UI. This should usually be another certificate issuer. In my case on local testing only, I setup a single-user-provider as authentication mechanism and chose `admin` as the username, so in the registry, I had to create a user called `admin` which I granted an access policy to the bucket I wanted to be able to read.

Only with both of the users setup, I was able to see my bucket listed in the NiFi ui and start version control.

avatar
Super Mentor

@Theoo 

Nice job on your path to solving the authorization issues, but you left out a few pieces

- When NiFi cluster nodes or a standalone NiFi instance communicates with a secured NiFi-Registry that communication MUST be authenticated and authorized in the NiFi-Registry side.  The established connection between NiFi and NiFi-Registry only supports authentication via a mutual TLS handshake (Client is identified via the certificate shared to the NiFi-Registry from NiFi).  Both NiFi and NiFi-Registry have identity mapping properties that can be added to the nifi-properties/nifi-registry.properties file that are used to manipulated the DN that comes from the client certificate.   

For example a NiFi host certificate with DN of "CN=nifi-node01, OU=NIFI" could be manipulated so the client string is only "nifi-node-01".

Users and NiFi nodes/instances to both NiFi and NiFi-Registry are just clients, there is no distinction between the two.  What matters is what each client is uniquely authorized to do within each service.

Whatever the client string happens to be, The NiFi nodes/instance must be authorized for the following global policies in NiFi-Registry:

"Can proxy user requests" (/proxy) with "Read, Write, and Delete" - This allows the NiFi nodes/instance to proxy some request made by the user authenticated in NiFi to perform some authorized request against NiFi-Registry (start version control, commit a new version of a version controlled Process Group (PG), etc.) since the NiFi user is not actually authenticating in to NiFi-Registry from NiFi. This does mean that the NiFi user string must exist as a user in NiFi-Registry and be authorized for the action they are trying perform.

"Can Manage Buckets" (/buckets) with "Read" - This policy is needed by the NiFi nodes/instance so that the NiFi background thread that occasionally communicates with NiFi-Registry to see if newer version of a version controlled PG is available or so NiFi can display a list of available buckets).  This request is not done on behalf of the user authenticated into NiFi.

MattWho_0-1633365611601.png

----
When it comes to the NiFi user, the policies needed in NiFi-Registry vary based on what you want that user to be able to do through NiFi or directly via the NiFi-Registry UI.

In order for a user who is currently authenticated and authorized in to NiFi to interact with NiFi-Registry, that user string would need to be authorized in NiFi-Registry for the following:
- A NiFi-Registry admin user would need to create a bucket and authorize the NiFi user on that bucket so it can be used by the NiFi user.
            - "READ" on the bucket would allow the user to import and existing version controlled flow from Nifi-Registry on the NiFi UI.
             - "WRITE" on the bucket would allow the user to start version control or change the version of the versioned PG in NiFi.
              - "Delete" on the bucket would allow a user who can authenticate in to NiFi-Registry to delete flows within that bucket. 

MattWho_1-1633365681781.png

--------

As far as authentication of users in to NiFi and/or NiFi-Registry, you can create certificates for each fo your users, but the most commonly used method is LDAP/AD based authentication.  You can add users in NiFi-Registry's authorizer so that those user string can be associated to authorization policies without those user even being able to authenticate and be authorized directly in to the NiFi-Registry's UI.  They simply need to exist for the proxied request that come from NiFi on that user's behalf.

Hope this exposes all that is needed in this thread.

Thanks,
Matt

avatar
Super Mentor

@JelenaS 

Making a bucket public only only control whether an unauthenticated and authorized user can import flows from a bucket on to the NiFi instance. So ONLY make it public if you want anonymous users to be able to use your version controlled flows in that bucket.  Users must still be authenticated and authorized in order to commit new flows to a public bucket.

As far as the global polices you set up for your "CN=<domainname>.net, OU=NiFi", that looks correct (don't need "write" on buckets), but is only correct if that string matches exactly what is coming from the certificates used on your secured NiFi instance(s) post and identity mapping happening on the NiFi-Registry server.  So check your nifi-registry.properties file for any configured Identity Mapping Properties:
https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#identity-mapping-prop...
For example:

nifi.registry.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?)$
nifi.registry.security.identity.mapping.value.dn=$1
nifi.registry.security.identity.mapping.transform.dn=NONE

with above and "CN=<domainname>.net, OU=NiFi", the string that would get checked for authorization in NiFi-Registry would be only "<domainname>.net" and thus be the string that would need to be authorized in instead of the full DN.  
When you are authenticated in to your NiFi instance as your nifi_admin user, what exact string is displayed in the upper right corner of the NiFi UI?  Is it "nifi_admin" or "CN=nifi_admin, OU=NiFi" because whatever displayed there is going to be the exact user string that gets proxied to the NiFi-Registry.

Also keep in mind that USER/CLIENT strings are case sensitive in both NIFi and NiFi-Registry.  Mapping transforms can be used to convert strings to all uppercase (UPPER) or all lowercase (LOWER).

Hope this helps,
Matt