Support Questions

Find answers, ask questions, and share your expertise

How to integrate Nifi with LDAP by using Ranger policy.

Ranger is integrated with LDAP and able to login Ranger UI through domain user.

I've made changes as per
https://pierrevillard.com/2017/01/24/integration-of-nifi-with-ldap/
But still I'm unable to get login

below is the configurations:

Template for login-identity-providers.xml

 <provider>
<identifier>ldap-provider</identifier>
<class>org.apache.nifi.ldap.LdapProvider</class>
<property name="Authentication Strategy">LDAPS</property>
<property name="Manager DN">uid=nifiadmin,ou=nifi,dc=abc,dc=com</property>
<property name="Manager Password">changeitambari</property>
<property name="TLS - Keystore">/usr/hdf/current/nifi/conf/keystore.jks</property>
<property name="TLS - Keystore Password">changeitambari</property>
<property name="TLS - Keystore Type">JKS</property>
<property name="TLS - Truststore">/usr/hdf/current/nifi/conf/truststore.jks</property>
<property name="TLS - Truststore Password">changeitambari</property>
<property name="TLS - Truststore Type">JKS</property>
<property name="TLS - Client Auth">REQUIRED</property>
<property name="TLS - Protocol">TLS</property>
<property name="TLS - Shutdown Gracefully"></property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldaps://ldap-dallas.abc.com:636</property>
<property name="User Search Base">cn=nifiadmin,ou=nifi,dc=abc,dc=com</property>
<property name="User Search Filter">sAMAccountName={nifiadmin}</property>
<property name="Authentication Expiration">12 hours</property>
</provider>
18 REPLIES 18

Master Guru
@suraj l

Is NiFi displaying the login screen or just blocking access to the UI?

If you are seeing a login screen, what do you see in the nifi-user.log after supplying your username and password?

Thanks,

Matt

Master Guru

@suraj l

Sorry, I am not following what you mean by "It is auto login through anonymous user"

Nifi follows the following Authentication order:

1. Server requests user/client certificate --> if client present valid trusted certificate then it is what will be used to authenticate
*** If above fails the rest of the methods must be configured in NiFi before they will be tried in this order.
2. SPNEGO --> if client presents a user principal via the http connection and it is validated, it will be used for authentication.
3. Either ldap-provider or kerberos-provider --> If both the above fail and a login-identity-provider has been configured, the user will be presented with a login screen which will be used for user authentication.

In your case, you appear to be using LDAPS with in the ldap-provider.
I am curious about your configured "User Search Base" and "User Search Filter" as they appear to be specific to just the nifiadmin user. In that case the the LDAPS search would only ever be able to return credentials for nifiadmin user. If you are trying to login in with some other user, that would explain the "The supplied username and password are not valid".

The next question is what method are you using for user authorization (this occurs after successful authentication.
Options include either NiFi's built-in file based authorizer or an external service like RANGER.

With the default file-based authorizer, you should have configured an "initial admin identity" and "node identities" for each node in your NiFi cluster.

You are saying you actually see the canvas which means some form of authentication and authorization is being successful.

The user log snippet you shared only shows successful authentication for what appears to be the DN for a nifi node? Di you load your nodes cert in to your browser per chance? If not did you authorize "anonymous" in your users.xml and authorizations.xml files?

thanks,

Matt

Is NiFi displaying the login screen or just blocking access to the UI?
>> Nifi displaying login screen. It is not blocking UI
It is auto login through anonymous user.

in logs :

2017-09-27 15:21:32,611 INFO [NiFi Web Server-597] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=ip-10-248-13-29.ec2.internal, OU=NIFI
2017-09-27 15:21:43,402 INFO [NiFi Web Server-547] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for anonymous




Also when I'm trying to log in through nifiadmin or domain user

            The supplied username and password are not valid

Hi Matt,

@Matt
As per your suggestion I have made changes in configurations but still I'm facing the same issue.

Here is the brief information about complete cluster setup:
I'm using HDF 3.0.1.0, Ambari 2.5.1, Nifi 1.2.0, zookeeper 3.4.6, Ranger 0.7.0, ambari metrics 0.1.0.
In this setup I've used total 9 nodes(3-Nifi, 3-zk,1-Ranger,1-db,1-Ambari). Everything is working fine. I've created policy in ranger and through that Nifi UI is accessible. Client certificate is used to access Nifi UI through browser.

Now requirement is,

Nifi and Ranger is need to authenticate with Ldap.

Ranger is authenticated with ldap (still it is not working well with ldaps).

Also Need to integrate Nifi with Ldap.

Currently,

Nifi UI is accessible through client certificate. But I wanted to use ldaps.

So I've disabled client certificate authentication in browser but then Nifi UI is directly login with anonymous user instead of giving login page. I'm struggling to identify the mistake from my configuration. I've attached screenshot of Nifi UI.

Please suggest necessary changes required in this setup.

authorizers.xml

<property name="Node Identity 1">CN=ip-Node1.ec2.internal, OU=NIFI</property>
<property name="Node Identity 2">CN=ip-Node2.ec2.internal, OU=NIFI</property>
<property name="Node Identity 3">CN=ip-Node3.ec2.internal, OU=NIFI</property>


users.xml and Template for login-identity-providers.xml

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<tenants>
    <groups/>
    <users>
        <user identifier="xxxxencrypted-codexxx" identity="CN=nifiadmin, OU=NIFI"/>
    </users>
</tenants>
<provider>
<identifier>ldap-provider</identifier>
<class>org.apache.nifi.ldap.LdapProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">uid=nifiadmin,ou=nifi,dc=abc,dc=com</property>
<property name="Manager Password">changeitambari</property>
<property name="TLS - Keystore">/usr/hdf/current/nifi/conf/keystore.jks</property>
<property name="TLS - Keystore Password">changeitambari</property>
<property name="TLS - Keystore Type">JKS</property>
<property name="TLS - Truststore">/usr/hdf/current/nifi/conf/truststore.jks</property>
<property name="TLS - Truststore Password">changeitambari</property>
<property name="TLS - Truststore Type">JKS</property>
<property name="TLS - Client Auth"></property>
<property name="TLS - Protocol">TLS</property>
<property name="TLS - Shutdown Gracefully"></property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://ldap.abc.com:3268</property>
<property name="User Search Base">ou=nifi,dc=abc,dc=com</property>
<property name="User Search Filter">uid={0}</property>
<property name="Authentication Expiration">12 hours</property>
</provider>

Nifi-user.log

2017-09-28 07:52:01,465 WARN [main] o.a.n.a.util.IdentityMappingUtil Identity Mapping property nifi.security.identity.mapping.pattern.dn was found, but was empty
==> nifi-user.log <==
2017-09-28 08:01:11,818 INFO [NiFi Web Server-16] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi.. Returning Conflict response.
2017-09-28 08:01:14,101 INFO [NiFi Web Server-21] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<><CN=ip-10-248-13-29.ec2.internal, OU=NIFI><CN=ip-10-248-12-214.ec2.internal, OU=NIFI>) GET https://ip-10-248-13-29.ec2.internal:8443/nifi-api/flow/current-user (source ip: 10.248.12.214)
2017-09-28 08:01:14,150 INFO [NiFi Web Server-21] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for anonymous


nifi-login-anonymous.png

Thank you,
Suraj

Master Guru

@suraj l

There are a few questions that need to be asked here....

You stated that you installed HDF NiFi and Ranger services. Please provide the current values assigned to the following properties in the nifi.properties file:

nifi.security.user.authorizer=
nifi.security.user.login.identity.provider=

It also may be helpful if you provided your login-identity-providers.xml file and you authorizers.xml file.

Based on your nifi-user.log above a user was never determined during the authentication phase "Attempting request for(<><CN=ip-10-248-13-29.ec2.internal, OU=NIFI><CN=ip-10-248-12-214.ec2.internal, OU=NIFI>)" This breakdown into "<>" <-- user, "<CN=ip-10-248-13-29.ec2.internal, OU=NIFI>" <--nifi node, "<CN=ip-10-248-12-214.ec2.internal, OU=NIFI>)" <-- "NiFi cluster coordinator node. As you can see the user was empty so it is being treated as anonymous.

So Anonymous is what is being passed to the authorizer. Question here is who really is your currently configured authorizer? NiFi's file based authorizer or Ranger?

The file-based authorizer uses the users.xml and authorizations.xml files of which you provided and I do not see anonymous in there anywhere. But if you are using Ranger, these files are not used anyway.

If you are using Ranger, do you have a policy configured that would allow anonymous access?

Thanks,

Matt

@Matt Clarke

1. nifi.properties

nifi.security.user.authorizer=ranger-provider
nifi.security.user.login.identity.provider=ldap-provider

2.login-identity-providers.xml

root@ip-10-248-13-29 conf]# cat /etc/nifi/3.0.1.1-5/0/login-identity-providers.xml





            'Referral Strategy' - Strategy for handling referrals. Possible values are FOLLOW, IGNORE, THROW.
            'Connect Timeout' - Duration of connect timeout. (i.e. 10 secs).
            'Read Timeout' - Duration of read timeout. (i.e. 10 secs).


            'Url' - Url of the LDAP servier (i.e. ldap://<hostname>:<port>).
            'User Search Base' - Base DN for searching for users (i.e. CN=Users,DC=example,DC=com).
            'User Search Filter' - Filter for searching for users against the 'User Search Base'.
            (i.e. sAMAccountName={0}). The user specified name is inserted into '{0}'.


            'Authentication Expiration' - The duration of how long the user authentication is valid
            for. If the user never logs out, they will be required to log back in following
            this duration.
            -->
            <provider>
  <identifier>ldap-provider</identifier>
  <class>org.apache.nifi.ldap.LdapProvider</class>
  <property name="Authentication Strategy">SIMPLE</property>
  <property name="Manager DN">ou=nifi,dc=abc,dc=com</property>
  <property encryption="aes/gcm/256" name="Manager Password">gQvcA6kcKRmaSvOa||4uR1XaOGmnOf9MsmTVfZaBehoyiUF+4/6QimJtTj</property>
  <property name="TLS - Keystore">/usr/hdf/current/nifi/conf/keystore.jks</property>
  <property encryption="aes/gcm/256" name="TLS - Keystore Password">XAWdj01lwKsP7V2m||iKQGPUjmLpLWkgCks5vaoNSI/dlOtbzGXiAuVcJV</property>
  <property name="TLS - Keystore Type">JKS</property>
  <property name="TLS - Truststore">/usr/hdf/current/nifi/conf/truststore.jks</property>
  <property encryption="aes/gcm/256" name="TLS - Truststore Password">bSd27hfNQARojrvf||1AlBXmZ4Fpp8vbNoEO2eywaRR5a1ptv6oThnf6Kd</property>
  <property name="TLS - Truststore Type">JKS</property>
  <property name="TLS - Client Auth"/>
  <property name="TLS - Protocol">TLS</property>
  <property name="TLS - Shutdown Gracefully"/>
  <property name="Referral Strategy">FOLLOW</property>
  <property name="Connect Timeout">10 secs</property>
  <property name="Read Timeout">10 secs</property>
  <property name="Url">ldap://ldap.abc.com:3268</property>
  <property name="User Search Base">dc=abc,dc=com</property>
  <property name="User Search Filter">uid{0}</property>
  <property name="Authentication Expiration">12 hours</property>
</provider>


            <!--
            Identity Provider for users logging in with username/password against a Kerberos KDC server.


            'Default Realm' - Default realm to provide when user enters incomplete user principal (i.e. NIFI.APACHE.ORG).
            'Authentication Expiration' - The duration of how long the user authentication is valid for. If the user never logs out, they will be required to log back in following this duration.
            -->


            <!-- To enable the kerberos-provider remove 2 lines. This is 1 of 2.


            <provider>
            <identifier>kerberos-provider</identifier>
            <class>org.apache.nifi.kerberos.KerberosProvider</class>
            <property name="Default Realm">None</property>
            <property name="Authentication Expiration">12 hours</property>
            </provider>


            To enable the kerberos-provider remove 2 lines. This is 2 of 2. -->




            </loginIdentityProviders>[root@ip-10-248-13-29 conf]#



3. I'm using Ranger. I've given access to below users for that Nifi Policy.

admin, keyadmin, {users}, {owner}, rangeradmin, amb_ranger_admin, rangertagsync

Is there anything, I can do changes in Ranger or edit Policy to get the login page on Nifi UI?

Hello,
Please suggest

@Matt Clarke @Pierre Villard

Latest changes:

authorizers.xml

<authorizer>
        <identifier>ranger-provider</identifier>


        <class>org.apache.nifi.ranger.authorization.RangerNiFiAuthorizer</class>
        <property name="Ranger Audit Config Path">/usr/hdf/current/nifi/conf/ranger-nifi-audit.xml</property>
        <property name="Ranger Security Config Path">/usr/hdf/current/nifi/conf/ranger-nifi-security.xml</property>
        <property name="Ranger Service Type">nifi</property>
        <property name="Ranger Application Id">nifi</property>
        <property name="Allow Anonymous">true</property>
        <property name="Ranger Admin Identity"></property>


        <property name="Ranger Kerberos Enabled">false</property>








<!-- Provide the identity (typically a DN) of each node when clustered (see tool tip for detailed description of Node Identity). Must be specified when Ranger Nifi plugin will not be used for authorization. -->


<property name="Node Identity 1">CN=ip-10-248-13-22.ec2.internal, OU=NIFI</property>
<property name="Node Identity 2">CN=ip-10-248-13-214.ec2.internal, OU=NIFI</property>
<property name="Node Identity 3">CN=ip-10-248-13-29.ec2.internal, OU=NIFI</property>


login-identity-providers.xml

      <provider>
  <identifier>ldap-provider</identifier>
  <class>org.apache.nifi.ldap.LdapProvider</class>
  <property name="Authentication Strategy">LDAPS</property>
  <property name="Manager DN">CN=hadoop_prd,OU=Service Accounts,OU=Hadoop,OU=CORE,OU=Servers and Services,DC=abc,DC=com</property>
  <property encryption="aes/gcm/256" name="Manager Password">xxx</property>
  <property name="TLS - Keystore">/usr/hdf/current/nifi/conf/keystore.jks</property>
  <property encryption="aes/gcm/256" name="TLS - Keystore Password">xx</property>
  <property name="TLS - Keystore Type">JKS</property>
  <property name="TLS - Truststore">/usr/hdf/current/nifi/conf/truststore.jks</property>
  <property encryption="aes/gcm/256" name="TLS - Truststore Password">xx</property>
  <property name="TLS - Truststore Type">JKS</property>
  <property name="TLS - Client Auth">WANT</property>
  <property name="TLS - Protocol">TLS</property>
  <property name="TLS - Shutdown Gracefully"/>
  <property name="Referral Strategy">FOLLOW</property>
  <property name="Connect Timeout">10 secs</property>
  <property name="Read Timeout">10 secs</property>
  <property name="Url">ldap://ldap.xxx.com:3268</property>
  <property name="User Search Base">dc=xxx,dc=com</property>
  <property name="User Search Filter">(cn={0})</property>
  <property name="Authentication Expiration">12 hours</property>
</provider>


Still its not working. users tab is not showing in Nifi UI at burger menu. Screenshot is attached. Please suggest.

39677-nifi-users.png

@Matt Clarke @Pierre Villard

Can you please suggest on above scenarios ?

Master Guru
@suraj l

* HCC Hint: try to avoid responding to an answer with another answer. Instead, respond to an answer by clicking "Add comment" under the answer.

When using an external authorizer such as Ranger, the NiFi UI will no longer show the "users" or "Policies" icons or menu items.

I am confused by your login-identity-providers.xml file configuration. Specifically,

<propertyname="Authentication Strategy">LDAPS</property>    

It is set to LDAPS; however, ...

<propertyname="Url">ldap://ldap.xxx.com:3268</property>

LDAPS requires that all communications are directly to secure end-point, so URL should start with ldaps://

Is it possible that your LDAP server is really setup for START_TLS? It is very similar to LDAPS (LDAPS was deprecated in favor of START_TLS) except initial connection if to ldap:// and then redirected to ldaps://

Try setting the Authentication Strategy to "START_TLS" and see if that works.

Thanks,

Matt

@Matt Clarke

Thanks for the suggestion about comments.

I have tried with START_TLS and ldaps:// url as well but still not working.

I have tried to connect ldap with SIMPLE option as well. But still Nifi is not syncing with LDAP.

How I can connect Nifi UI through LDAP/AD users?

Can you please suggest on below configuration:

  <provider>
            <identifier>ldap-provider</identifier>
            <class>org.apache.nifi.ldap.LdapProvider</class>
            <property name="Identity Strategy">USE_DN</property>
            <property name="Authentication Strategy">SIMPLE</property>
            <property name="Manager DN">CN=hadoop_prd_ad_user,OU=Service 
             Accounts,OU=Hadoop,DC=abc,DC=com</property>
            <property name="Manager Password">xxx</property>
            <property name="Referral Strategy">FOLLOW</property>
            <property name="Connect Timeout">10 secs</property>
            <property name="Read Timeout">10 secs</property>
            <property name="Url">ldap://ldap.abc.com:389</property>
            <property name="User Search Base">DC=abc,DC=com</property>
            <property name="User Search Filter">sAMAccountName={0}</property>
            <property name="Authentication Expiration">12 hours</property>
            </provider>

Master Guru
@suraj l

NiFi does not sync with LDAP. LDAP is simply used for user authentication.

You said you get the login screen, but once you enter a valid username and password you get logged in as anonymous?

Or you never see login screen and you are logged in as anonymous automatically?

What get returned if you run a ldapsearch command from your NiFi server?

ldapsearch -x -D "CN=hadoop_prd_ad_user,OU=Service Accounts,OU=Hadoop,DC=abc,DC=com" -W -b "DC=abc,DC=com"

Does the returned listing include your nifiadmin or domain users?

Thanks,

Matt

@Matt Clarke

In Nifi UI, It is automatically logged in through anonymous user. But at the top there is option to log in through different user. Please check screenshot attached. I tried to login with another user but doesn't work.

41510-nifi-log-in.png

with your ldapsearch command gives below output.

ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

Please check below output as well

[root@ip-x ~]#ldapsearch -h ldap.abc.com -p 389 -D "CN=hadoop_prd_ad_user,OU=Service Accounts,OU=Hadoop" -b "DC=abc,DC=com"
# extended LDIF
#
# LDAPv3
# base <DC=abc,DC=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#


# search result
search: 2
result: 1 Operations error
text: 000004DC: LdapErr: DSID-0C0907C2, comment: In order to perform this opera
 tion a successful bind must be completed on the connection., data 0, v2580


# numResponses: 1

[root@ip-x ~]#
<br>

Port 389 is open and telnet is working fine from Nifi server to LDAP server. It doesn not return with nifiadmin or domain users.

What could be the reason?

How can I login Nifi UI with ldap user?

Thanks,

Suraj

@Matt Clarke

I have copied keystore.jks and truststore.jks files from Nifi server to Ranger policy and Now I'm able to do login Nifi UI with LDAP users.

My next challenge is I wanted to do user authentication with LDAPS. In Ranger UI I'm able to do log in with LDAP authentication but when I do the changes for LDAPS, it doesn't work. Can you please suggest

Thank you.

Master Guru

@suraj l

I am not an expert on Ranger. Since your NiFi piece is now working, I suggest accepting an answer in this thread and starting a new HCC post around enabling LDAPS within Ranger. Other community members who are Ranger experts are unlikely to see this latest question.

Thanks,

Matt

@Matt Clarke Thank you

I found the solution. Issue is fixed now.

In my case, one of LDAP username is 'dvteam' but in LDAP database there was full description of username as 'architecture dev team, locations, team details, etc'.

Error messages I found in nifi-user.log. is 'architecture dev team' user was trying to authenticate with nifi nodes. Authentication was successful but authorizations not happening.

The username which I've mentioned in initial admin identity was 'dvteam'.(cn=dvteam,ou=xx,ou=xx,ou=xx,ou=xx,dc=abc,dc=com) Then as per logs, I changed it to (cn=architecture dev team,ou=xx,ou=xx,ou=xx,ou=xx,dc=abc,dc=com)

Also there was some mismatch about host names in node identities section. 'hostname -f' shows a hostname ip-zz-xx-ec2-internal. So, I have given 'ip-zz-xx-ec2-internal' in node identities section but that was not working. Then I have changed the hostnames to 'nifi1.abc.local' and mentioned in node identities.

In 'Template for login-identity-providers.xml' I've made some changes. Earlier I had set 'use_username' in '<property name="Identity Strategy">USE_DN</property>' this section.

later I've changed to use_dn. because as per nifi-user log authentication is happening with LDAP user 'architecture dev team'.

So in my case user_username was not working for authentications.

Every configurations changes I used to remove authorizations.xml and users.xml file from my all nifi nodes.

Also There was confusion on about 'OU' in Node identities section.

What does it mean OU in node identities section? I don't know yet.

Later I've mentioned 'OU=nifi' and also gave host names as 'nifi1.abc.local' , 'nifi2.abc.local', etc.

I have added AD/LDAP user in Initial Admin Identity(cn=architecture dev team,ou=xx,ou=xx,ou=xx,ou=xx,dc=abc,dc=com)

After setting above all, I was facing an error about setting nifi.security.identity.mapping.pattern.dn.

There was a challenge about the pattern definition.

There was 4 'ou' I have defined in initial admin identities and login-identity-providers.xml.

So I've used below pattern and it worked well.

^cn=(.?),ou=(.?),ou=(.?),ou=(.?),ou=(.?),dc=(.?),dc=(.?)$

Note: I have removed Ranger completely.

Thanks,

Suraj

New Contributor

Hey Suraj, so you were able to integrate LDAP with NIFI and manage users with LDAP right? Ranger is completely out of picture. You are not using it for authorization.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.