Getting Unable to obtain listing of buckets: org.apache.nifi.registry.client.NiFiRegistryException: Error retrieving all buckets Error while "start version control" in NiFi

[#1]

AM using HDF 3.1.0 with NiFi and Nifi Registry. When i try use start version control from NiFi processors, am ending up getting Unable to obtain listing of buckets: org.apache.nifi.registry.client.NiFiRegistryException:

[#2 · Accepted solution]

@Bala S

I've seen a similar error:

Unable to obtain listing of buckets: java.net.ConnectException: Connection refused

when the URL of the Registry Client (Under the top-right Global Menu-->Controller Settings-->Registry Clients tab) is incorrect.

Is there more in your Registry log (nifi-registry-app.log) Is your NiFi Registry secured? Other things to check:

-that you have buckets in your Registry

-that your user has privilege to access the buckets in your Registry

I wrote a couple articles regarding the Registry that might be useful:

https://community.hortonworks.com/articles/166802/versioned-dataflows-with-apache-nifi-15-and-apache...

https://community.hortonworks.com/articles/170966/setting-up-a-secure-apache-nifi-registry.html

https://community.hortonworks.com/articles/171173/setting-up-a-secure-nifi-to-integrate-with-a-secur...

[#3]

@Andrew Lim

Below are the lines from the nifi-registry-app.log,

2018-02-18 16:46:01,149 INFO [NiFi Registry Web Server-36] o.a.n.r.w.m.ResourceNotFoundExceptionMapper org.apache.nifi.registry.exception.ResourceNotFoundException: No policy found for action='read', resource='/buckets/1f76510d-c48e-4ee6-b883-7edfcfe57e40'. Returning Not Found response. 2018-02-18 16:46:01,239 INFO [NiFi Registry Web Server-30] o.a.n.r.w.m.ResourceNotFoundExceptionMapper org.apache.nifi.registry.exception.ResourceNotFoundException: No policy found for action='write', resource='/buckets/1f76510d-c48e-4ee6-b883-7edfcfe57e40'. Returning Not Found response. 2018-02-18 16:46:01,331 INFO [NiFi Registry Web Server-15] o.a.n.r.w.m.ResourceNotFoundExceptionMapper org.apache.nifi.registry.exception.ResourceNotFoundException: No policy found for action='delete', resource='/buckets/1f76510d-c48e-4ee6-b883-7edfcfe57e40'. Returning Not Found response.

s your NiFi Registry secured? Yes

-that you have buckets in your Registry - Yes. have created one bucket

-that your user has privilege to access the buckets in your Registry - Yes, I have assigned new policy for the bucket created

Couple of questions.

1) Where do you run your nifi-registry service? on Ambari server or NiFi Nodes?

2) Do we need to specify Node identity while configuring nifi-registry?

[#4]

@Bala S

I'm sorry I didn't realize this earlier, but I believe I know the cause of your issue as I have reproduced it.

You need to configure a localhost user in your registry. I covered this in the section "Localhost User Configuration in Registry" in this HCC article:

https://community.hortonworks.com/articles/171173/setting-up-a-secure-nifi-to-integrate-with-a-secur...

Localhost user needs to have the following privileges:

-Read to all buckets is necessary because NiFi itself checks the status of all the flows in NiFi to determine if they are in sync with the Registry.  For example, checking whether a new version of flow is available for upgrade.

-Proxy privileges are needed because when versioned flow actions occur, NiFi proxies the end user over to the Registry.  For example, when saving a version to the Registry it will be done as the actual user who performed this action in NiFi.

If this solves your problem, please accept this as answer so others running into it can find this solution. Thanks!

[#5]

@Louis Allen

I believe you have found your problem in your NiFi node certificates:

-

ExtendedKeyUsages [ serverAuth ]

-

The NIFi nodes are establishing the connection to the the NIFi-registry to retrieve bucket information. That means NiFi is acting as a client and not a server in the TLS 2-way handshake. Since NiFi nodes have no client certificate to offer to NiFi-Registry for the purpose of authentication and then authorization, the nodes are failing to retrieve the bucket listing.
-

NiFi certs must support both clientAuth and serverAuth. NiFi registry is just one example of why this is important. Using NiFi's Site-To-Site capability via Remote Process Groups (RPGs) is another example of when clientAuth would be required to send data between to secured NiFis.

-

You are going to need to generate new certificates for all your NIFi nodes that support both ClientAuth and ServerAuth.

-

Thank you,

Matt

-

If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.

[#6]

Hello.

I have try to find information about how to generate new certificates that support ClientAuth and ServerAuth and I did not find something about it.

Could You explain me how to do it?

Thanks.

"You are going to need to generate new certificates for all your NIFi nodes that support both ClientAuth and ServerAuth."

[#7]

I'm currently having the same issue and need some help if possible...

I have generated all certs using our internal CA (not the tls-toolkit) and have been working with NiFi successfully now for a few weeks. We are only now looking at integrating NiFi Registry and have run into some issues.

I have:

• Added all certs into relevant truststores
• Added the nodes into initial users in authorizers.xml on Registry and given them full permissions
• Added the users that we are logging in as on NiFi to NiFi registry and given them full permissions
• Created a bucket and added the users/nodes

I'm getting the following error messages:

Unable to obtain listing of buckets: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

or

Unable to obtain listing of buckets: java.net.SocketException: Broken pipe (Write failed)

[#8]

@Louis Allen

The Error "Failed to connect node to cluster because local flow controller partially updated." indicates you are having an authentication issue and not an authorization issue (yet). When communicating with a secured Registry, a 2-way TLS authentication is required between NiFi and NiFi-registry.

-

Use the following command to get the full certificate chain from NiFi and NiFi-registry:

#openssl s_client -connect <nifi-node>:<nifi-port>|grep -A 10 chain

#openssl s_client -connect <nifi-registry>:<registry-port>|grep -A 10 chain

10 lines should be more then enough to get full chain, but if not you may need to increase this number.

-

Things to verify on NiFi side:

1. Perform a verbose listing on the keystore used by each nifi-node and verify that each contains only a single "PrivateKeyEntry" and the "PrivateKeyEntry" has an ExtendedKeyUsage that supports both client_auth and serverAuth.

2. Perform a verbose listing on the truststore used by each nifi-node and verify the ever CA returned by the openssl command run against the registry is found as a "TrustedCertEntry". (this may be 1 or more intermediate CAs and a root CA)

-

Things to verify on NiFi-Registry side:

1. Perform a verbose listing on the truststore used by the nifi-registry and verify the ever CA returned by the openssl command run against each NiFi node is found as a "TrustedCertEntry". (this may be 1 or more intermediate CAs and a root CA)

-

Each NiFi node should appear in the your list of users and been granted both the "Can proxy user requests" and "READ" access on the "Can manage buckets" policies. If you have not setup and identity mapping patterns in NiFi-registry, the NiFi nodes user names will be the complete node DN.

-

Thank you,

Matt

[#9]

Hi Matt,

Thank you for taking the time to reply/help.

Certificate chains look ok to me.

On the NiFi nodes when I output the keystore information I get the following under ExtendedKeyUsage:

ExtendedKeyUsages [ serverAuth ]

On the NiFi nodes when I output the truststore information I get the following for the root and intermediate CA's under Entry type:

Entry type: trustedCertEntry

On the NiFi Registry node when I output the truststore information I get the following for the root and intermediate CA's under Entry type:

Entry type: trustedCertEntry

I have setup NiFi Registry to allow users to login using LDAP (including group sync) and have setup a composite-configurable-user-group-provider

<userGroupProvider>
<identifier>composite-configurable-user-group-provider</identifier><class>org.apache.nifi.registry.security.authorization.CompositeUserGroupProvider</class>
<property name="User Group Provider 1">file-user-group-provider</property>
<property name="User Group Provider 2">ldap-user-group-provider</property>
</userGroupProvider>

So that I have the ability to add the NiFi nodes under the file-user-group-provider

<userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property> <property name="Legacy Authorized Users File"></property> <property name="Initial User Identity 1">###hidden full dn###</property>
<property name="Initial User Identity 2">###hidden full dn###</property>
<property name="Initial User Identity 3">###hidden full dn###</property>
</userGroupProvider>

And within NiFi I have granted each node all permissions.

[#10]

@louis_allen 

NiFi node certificates must meet the following minimum criteria:

1. Extended Key usage (EKU) must support both "clientAuth" and "serverAuth".  Since yours only support "serverAuth" the certificate can not be used in a mutual TLS handshake where the NiFi node is acting as a client, such as when the NiFi instance/node is communicating with NiFi-Registry or another node in a multi-node NiFi cluster.

2. The NiFi and NiFi-Registry keystore can ONLY contain ONE "PrivateKeyEntry". There is no property to tell NiFi or NiFi-Registry which PrivateKeyEntry to use when multiples exists in the keystore.
3. SubjectAlternativeName (SAN) - At least one SAN entry is required that matches the hostname of the server on which the certificate is being used.
4. NiFi and NiFi-Registry do not support wildcards used in certificate DNs.


Once you have gotten new certificates for your NiFi instances that support "clientAuth", you'll need to make sure the DN from those certificates is properly authorized in NiFi-Registry to the two policies I outlined in my previous response (Read - buckets and proxy).

This will resolve your issue.

Hope this helps,

Matt