Trying to integrate Apache Nifi with Apache Ranger gives: Unable to retrieve any resources using given parameters. Status Code = 401

[#1]

I have been trying to integrate Nifi with Apache Ranger. When I manually configure policies in Ranger for Nifi the policies are fetched by Nifi and authorization works fine.  But when I try to define the service definition and test the connection its is giving the following error.

The configuration in service definition is shown below.

The authentication in nifi is setup by following this article.

Note: Only Nifi is setup in SSL mode. Apache Ranger doesn't have SSL mode.

What could be the possible reason this is happening?

[#2 · Accepted solution]

@pacman 

Seems odd to me that in the shared Ranger configuration you have configured a truststore where Ranger is expecting a keystore.  Then you have your truststore configured with the keystore instead of truststore.

The Ranger service is attempting to connect to NiFi's /resource  NiFi resource Identifier endpoint in order to obtain an up-to-date list of NIFi Resource Identifiers in order to simplify creating new policies in Ranger for NiFi.  The endpoint requires authorization and NiFi gets its authorization from the policies it downloads from Ranger.  So we have a which comes first, chicken or the egg, scenario here.   You will need to authorize the clientAuth certificate configure in the keystore for READ on the "/resources" NiFi Resource Identifier before Ranger will be able to fetch all the rest of the NiFi resource Identifiers.

Before that is possible, Ranger will need to be able to successfully negiotate a mutualTLS handshake with NiFi.  This does not require that Ranger server is SSL enabled, just that the NiFi service within Ranger is configured with a keystore and truststore capable of facilitating that mutualTLS handshake with the secured NiFi.

Also keep in mind that Ranger service user is executing this client connection to NiFi and thus the keystore and truststore configured must be accessible by the Ranger service user.  The Truststore used in Ranger NiFi service must contain the complete trust chain for all the NiFi cluster node's certificates.  The ClientAuth certificate in the keystore used in the Ranger NiFi service must be trusted by all the NiFi cluster nodes (meanings its complete trust chain must exist in NiFi's truststore configured in the nifi.properties file).

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3]

Looks like my issue is with specifying policy.download.auth.users as nifi. I was integrating pluggin following this article. I skipped the step 7 

• Give user and group ownership with nifi process user and set permission 400 to files ranger-nifi-audit.xml and ranger-nifi-security.xml

I tried adding a new user nifi and group nifi in my ubuntu and permission 400 and ownership to nifi user for files ranger-nifi-audit.xml and ranger-nifi-security.xml. But still its throwing 401.

Update:

I modified the bootstrap.conf file in Nifi to update run.as  property to nifi and other users as well. But still there isn't any effect.

@MattWho  @bbende  any thoughts on this?

[#4 · Accepted solution]

@pacman 

Seems odd to me that in the shared Ranger configuration you have configured a truststore where Ranger is expecting a keystore.  Then you have your truststore configured with the keystore instead of truststore.

The Ranger service is attempting to connect to NiFi's /resource  NiFi resource Identifier endpoint in order to obtain an up-to-date list of NIFi Resource Identifiers in order to simplify creating new policies in Ranger for NiFi.  The endpoint requires authorization and NiFi gets its authorization from the policies it downloads from Ranger.  So we have a which comes first, chicken or the egg, scenario here.   You will need to authorize the clientAuth certificate configure in the keystore for READ on the "/resources" NiFi Resource Identifier before Ranger will be able to fetch all the rest of the NiFi resource Identifiers.

Before that is possible, Ranger will need to be able to successfully negiotate a mutualTLS handshake with NiFi.  This does not require that Ranger server is SSL enabled, just that the NiFi service within Ranger is configured with a keystore and truststore capable of facilitating that mutualTLS handshake with the secured NiFi.

Also keep in mind that Ranger service user is executing this client connection to NiFi and thus the keystore and truststore configured must be accessible by the Ranger service user.  The Truststore used in Ranger NiFi service must contain the complete trust chain for all the NiFi cluster node's certificates.  The ClientAuth certificate in the keystore used in the Ranger NiFi service must be trusted by all the NiFi cluster nodes (meanings its complete trust chain must exist in NiFi's truststore configured in the nifi.properties file).

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#5]

@MattWho  the 401 issue was a silly issue with specifying truststore instead of keystore and viceversa. But now it is throwing SSL Handshake exeception as below.

Since both nifi and ranger is running on my local machine I configured the same truststore and keystore for both.

Update: 

Not sure what is wrong with the SSL certs I created using openSSL, but when I create certifcates with nifi tool kit the exception has gone. Now its throwing 403.

[#6 · Accepted solution]

@pacman 
The "unable to find valid certificate path to request target" is caused when trust can not be established in the TLS handshake.  This is the result of missing trusted cert entry(s) in the truststore (missing or incomplete trust chain).   When you used the toolkit to create your certficates, it also built the corresponding truststore with the required trust chain. 

The HTTP 403 exception indicated not authorized while the 401 indicated failed authentication.  
Authentication was determined through the mutualTLS handshake between the clientAuth certificate presented from the keystore configured in Ranger and serverAuth certificate presented by NiFi.   The DN from the clientAuth certificate is used as the user identity in the authentication (keep in mind that during authentication the final user identity is determined after evaluating the user identity returned during authentication against any configured Identity mapping patterns configured in the nifi.properties file.

If you inspect the nifi-user.log, you should see attempt to access the .../resources NiFi endpoint (This is endpoint where Ranger is trying to fetch the list of NiFi resource identifiers).  In that nifi-user.log you will see the actual post authentication user identity being used (full certificate DN or some valued mapped from that full DN).  This is where my early response about needing to authorize the ranger clientAuth certificate user identity in Ranger manually against the "/resources" NiFi resource Identifier is needed.
So you'll need to make sure the exact user identity from the nifi-user.log is added as a user in Ranger and added to a NiFi service policy in ranger giving that user authorization for "/resources".  Once NiFi downloads the updated policy json from Ranger, Ranger NiFi service client will now be authorized to fetch all NiFi Resource Identifiers from the /resources endpoint.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#7]

Thank you @MattWho for your valuable insights. Integrations are working fine now. But before we end this thread I have a  few more questions that you can possibly answer. 
1) Currently I have configured an SSL user and users from LDAP. When logging in browser during loading nifi the first option that comes in is sign in with the SSL user. Is there a way to disable it?

2) Currently I have added the SSL user as initial admin identity, then removed excessive permissions from authorizations.xml. 

Are there any best practice for achieving both?

Also I don't think apart from /flow other resource access are working properly. I added the resource paths for creating processors but doesn't seem to work.

[#8]

@pacman 
1. There is noway to disable TLS. If you remove your TLS certificate from your browser or use and incognito window your client certificate will not be presented in the TLS exchange. NiFi requires TLS certificate authentication for NiFi to NiFi authenticated and authorized connections (for example in a multi-node NiFi cluster or utilizing NiFi Site-to-Site between different NiFi deployments).  When NiFi is secured an NO other methods of authentication are configured, NiFi will "REQUIRE" a MutualTLS exchange.  Once at least one additional method of authentication is  configured, NiFi will "WANT" a client certificate and if one is not presented from the client, NiFi will move on to next authentication method.

2. I am not clear what you mean by "removed excessive permissions from authorizations.xml".  If you are using Ranger, the authorizations.xml file is not being used.  That file would have been created by the file-access-policy-provider.  Ranger does not use this provider.  There really is no concept of an "initial admin" when using Ranger.  You'll need to add authorization for what you need manually in Ranger.  The "Initial Admin" is used when NiFi authorization is handled by a local file provider so that a user can be setup on startup that has ability to access NiFi and setup additional authorizations from within the NiFi UI. 

I recommend starting a new community question so we don't make this thread overly complicated by solving many unrelated issues.   There are some NiFi Resource Identifiers that would only apply to file based authorization, but all other do work when used correctly.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt