Cannot got login page when i enable ssl & ldap in nifi registry with cloudera flow manager 1.0.1

[#1]

Hello,

I am working with cloudera flow manager version 1.0.1. I cannot got login page when i enable ssl & ldap in nifi registry instead of this page, and there is an node identity.  I have checked the config that may be correctly. and try more time such as remove /var/lib/nifiregister/* or /var/run/cloudera-scm-agent/process/***-nifiregistry-NIFI_REGISTRY_SERVER/* 

the blow pic shows the login user is node id.

the related ldap info is below, but there is not userDN config option in cloudera manager. there is same setting between nifi and nifi registry.

2019-11-02 10:24:59,167 INFO org.springframework.ldap.core.support.AbstractContextSource: Property 'userDn' not set - anonymous context will be used for read-write operations

The behavior is very strange. 

Who could help me what i missed?

Thanks,

Paul

[#2 · Accepted solution]

@Paul Yang 

A couple observation based on provided information:

1.  You have need clientAuth enabled

nifi.registry.security.needClientAuth=true

 With this enabled in NiFi-Registry the only authentication method supported will be 2-way TLS.  If you want to support other authentication methods lik Spnego, LDAP, and/or kerberos, this property must be false.

2. Your users.xml (used for user authorization and not authentication) only contains one user.  I am assuming the user is : 

CN=arch-fndtf04.beta1.fn, OU=NIFI

And your authorizations.xml also show that this is the only user that has been authorized to a bunch of policies.  Your ldap user you want to authorize must exist in the users.xml file.  My guess here is that you did not set the "Initial admin" user in your authorizers.xml (which you did not share).  There are two possible providers in the authorizers were you can set the initial admin (file-user-group-provider - Only set here if your initial admin user is NOT coming from the ldap-user-group-provider.  file-access-provider - initial admin must be set here so initial admin policies are created for this user.

3. The users.xml and authorizations.xml files are only generated if they do not already exist.  If you go back and add an initial admin to your authorizers.xml, you will need to delete/rename the existing users.xml and authorizations.xml files o new ones can be generated in startup.

4. Since need clientAuth was set to true and UI clearly shows that user string: 

CN=arch-fndtf04.beta1.fn, OU=NIFI

successfully authenticated, your browser must have this client certificate loaded and presented to the server when you navigated to the URL for your NiFi-Registry.  I really see no reason why this certificate was loaded int to your browser.  NiFi-Registry, even when need clientAuth is false, will always try mutual TLS authentication first (becomes a WANT instead of REQUIRE when need clientAuth is false).  If no client certificate is provided in TLS handshake, next auth method tried is Spnego (if configured), and finally a configured login provider from the identity-providers.xml.

Hope this info helps with correcting your setup issues.
Thanks,

Matt

[#3]

Update:  

the below is the log:

2019-11-03 20:06:10,531 DEBUG [NiFi Registry Web Server-19] o.a.n.r.w.s.a.IdentityFilter Attempting to extract user credentials using X509IdentityProvider
2019-11-03 20:06:10,531 DEBUG [NiFi Registry Web Server-19] o.a.n.r.w.s.a.IdentityFilter Adding credentials claim to SecurityContext to be authenticated. Credentials extracted by X509IdentityProvider: AuthenticationRequest{username='CN=arch-fndtf04.beta1.fn, OU=NIFI', credentials=[PROTECTED], details=null}
2019-11-03 20:06:10,531 DEBUG [NiFi Registry Web Server-19] o.a.n.r.w.s.a.IdentityFilter Credentials already extracted for [org.apache.nifi.registry.web.security.authentication.AuthenticationRequestToken$1@39a29a41], skipping credentials extraction filter using JwtIdentityProvider
2019-11-03 20:06:10,532 DEBUG [NiFi Registry Web Server-19] o.a.n.r.w.s.a.ResourceAuthorizationFilter Request filter authorization check is not required for this HTTP Method on this resource. Allowing request to proceed. An additional authorization check might be performed downstream of this filter.
2019-11-03 20:06:10,688 INFO [NiFi Registry Web Server-12] o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos service ticket login not supported by this NiFi Registry. Returning Conflict response.
2019-11-03 20:06:10,691 DEBUG [NiFi Registry Web Server-12] o.a.n.r.w.m.IllegalStateExceptionMapper 
java.lang.IllegalStateException: Kerberos service ticket login not supported by this NiFi Registry
	at org.apache.nifi.registry.web.api.AccessResource.createAccessTokenUsingKerberosTicket(AccessResource.java:285) ~[classes/:na]
......

2019-11-03 20:06:10,721 DEBUG [NiFi Registry Web Server-19] o.a.n.r.w.s.a.IdentityFilter Attempting to extract user credentials using X509IdentityProvider
2019-11-03 20:06:10,722 DEBUG [NiFi Registry Web Server-19] o.a.n.r.w.s.a.IdentityFilter Adding credentials claim to SecurityContext to be authenticated. Credentials extracted by X509IdentityProvider: AuthenticationRequest{username='CN=arch-fndtf04.beta1.fn, OU=NIFI', credentials=[PROTECTED], details=null}
2019-11-03 20:06:10,722 DEBUG [NiFi Registry Web Server-19] o.a.n.r.w.s.a.IdentityFilter Credentials already extracted for [org.apache.nifi.registry.web.security.authentication.AuthenticationRequestToken$1@2929ad59], skipping credentials extraction filter using JwtIdentityProvider
2019-11-03 20:06:10,723 DEBUG [NiFi Registry Web Server-19] o.a.n.r.w.s.a.ResourceAuthorizationFilter Request filter authorization check is not required for this HTTP Method on this resource. Allowing request to proceed. An additional authorization check might be performed downstream of this filter.
2019-11-03 20:06:10,784 DEBUG [NiFi Registry Web Server-19] o.a.n.r.w.s.a.IdentityFilter Attempting to extract user credentials using X509IdentityProvider
2019-11-03 20:06:10,784 DEBUG [NiFi Registry Web Server-17] o.a.n.r.w.s.a.IdentityFilter Attempting to extract user credentials using X509IdentityProvider
2019-11-03 20:06:10,784 DEBUG [NiFi Registry Web Server-19] o.a.n.r.w.s.a.IdentityFilter Adding credentials claim to SecurityContext to be authenticated. Credentials extracted by X509IdentityProvider: AuthenticationRequest{username='CN=arch-fndtf04.beta1.fn, OU=NIFI', credentials=[PROTECTED], details=null}
2019-11-03 20:06:10,784 DEBUG [NiFi Registry Web Server-17] o.a.n.r.w.s.a.IdentityFilter Adding credentials claim to SecurityContext to be authenticated. Credentials extracted by X509IdentityProvider: AuthenticationRequest{username='CN=arch-fndtf04.beta1.fn, OU=NIFI', credentials=[PROTECTED], details=null}
2019-11-03 20:06:10,784 DEBUG [NiFi Registry Web Server-19] o.a.n.r.w.s.a.IdentityFilter Credentials already extracted for [org.apache.nifi.registry.web.security.authentication.AuthenticationRequestToken$1@7bf82c3a], skipping credentials extraction filter using JwtIdentityProvider
2019-11-03 20:06:10,784 DEBUG [NiFi Registry Web Server-17] o.a.n.r.w.s.a.IdentityFilter Credentials already extracted for [org.apache.nifi.registry.web.security.authentication.AuthenticationRequestToken$1@69275fb3], skipping credentials extraction filter using JwtIdentityProvider
2019-11-03 20:06:10,785 DEBUG [NiFi Registry Web Server-19] o.a.n.r.w.s.a.ResourceAuthorizationFilter Request filter authorization check is not required for this HTTP Method on this resource. Allowing request to proceed. An additional authorization check might be performed downstream of this filter.
2019-11-03 20:06:10,785 DEBUG [NiFi Registry Web Server-17] o.a.n.r.w.s.a.ResourceAuthorizationFilter Request filter authorization check is not required for this HTTP Method on this resource. Allowing request to proceed. An additional authorization check might be performed downstream of this filter.

The below is my configurations:

nifi-registry.properties

nifi.registry.db.directory=
nifi.registry.db.driver.class=org.h2.Driver
nifi.registry.db.driver.directory=
nifi.registry.db.maxConnections=5
nifi.registry.db.password=UqZCvEAQeGvUUIGH||82ibCgtpV4JUhkFCnxQkW7kXxkmkHrc
nifi.registry.db.password.protected=aes/gcm/256
nifi.registry.db.sql.debug=false
nifi.registry.db.url=jdbc:h2:/var/lib/nifiregistry/database/nifi-registry-primary;AUTOCOMMIT=OFF;DB_CLOSE_ON_EXIT=FALSE;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
nifi.registry.db.url.append=
nifi.registry.db.username=nifireg
nifi.registry.initial.admin.identity=*******
nifi.registry.kerberos.krb5.file=/etc/krb5.conf
nifi.registry.kerberos.service.keytab.location=/var/run/cloudera-scm-agent/process/238-nifiregistry-NIFI_REGISTRY_SERVER/nifiregistry.keytab
nifi.registry.kerberos.spnego.authentication.expiration=12 hours
nifi.registry.kerberos.spnego.keytab.location=/var/run/cloudera-scm-agent/process/238-nifiregistry-NIFI_REGISTRY_SERVER/nifiregistry.keytab
nifi.registry.providers.configuration.file=/var/run/cloudera-scm-agent/process/238-nifiregistry-NIFI_REGISTRY_SERVER/providers.xml
nifi.registry.security.authorizer=managed-authorizer
nifi.registry.security.authorizers.configuration.file=/var/run/cloudera-scm-agent/process/238-nifiregistry-NIFI_REGISTRY_SERVER/authorizers.xml
nifi.registry.security.identity.provider=ldap-provider
nifi.registry.security.identity.providers.configuration.file=/var/run/cloudera-scm-agent/process/238-nifiregistry-NIFI_REGISTRY_SERVER/identity-providers.xml
nifi.registry.security.keyPasswd=cpDNEjgeOtHgUKBg||/TtGPhbQyltKWVvH9Cj7rj3ZVYZO
nifi.registry.security.keyPasswd.protected=aes/gcm/256
nifi.registry.security.keystore=/var/lib/nifiregistry/cert/keystore.jks
nifi.registry.security.keystorePasswd=QgccvlFai9XXLFUB||Pgu0W6X+BYYSPCiu1drPcqtWIru7
nifi.registry.security.keystorePasswd.protected=aes/gcm/256
nifi.registry.security.keystoreType=jks
nifi.registry.security.needClientAuth=true
nifi.registry.security.truststore=/var/lib/nifiregistry/cert/truststore.jks
nifi.registry.security.truststorePasswd=TKpFfRmNkxQD5xqg||IY8IZookjPjKpGiKiTplZpvmkMRB
nifi.registry.security.truststorePasswd.protected=aes/gcm/256
nifi.registry.security.truststoreType=jks
nifi.registry.sensitive.props.additional.keys=nifi.registry.db.password
nifi.registry.web.http.host=
nifi.registry.web.http.port=
nifi.registry.web.https.host=arch-fndtf03.beta1.fn
nifi.registry.web.https.port=18433
nifi.registry.web.jetty.threads=200
nifi.registry.web.jetty.working.directory=/var/lib/nifiregistry/work/jetty
nifi.registry.web.war.directory=/opt/cloudera/parcels/CFM-1.0.1.0/REGISTRY/lib

identity-providers.xml:

<identityProviders>

    <provider>
        <identifier>kerberos-identity-provider</identifier>
        <class>org.apache.nifi.registry.web.security.authentication.kerberos.KerberosIdentityProvider</class>
        <property name="Authentication Expiration">12 hours</property>
        <property name="Default Realm"></property>
        <property name="Enable Debug">false</property>
        
    </provider>

    <provider>
        <identifier>ldap-provider</identifier>
        <class>org.apache.nifi.registry.security.ldap.LdapIdentityProvider</class>
        <property name="User Search Base">***</property>
        <property name="Connect Timeout">10 secs</property>
        <property encryption="aes/gcm/256" name="Manager Password">**</property>
        <property name="Authentication Strategy">SIMPLE</property>
        <property name="Manager DN">**</property>
        <property name="Referral Strategy">FOLLOW</property>
        <property name="Identity Strategy">USE_USERNAME</property>
        <property name="User Search Filter">cn={0}</property>
        <property name="Authentication Expiration">12 hours</property>
        <property name="Read Timeout"></property>
        <property name="Url">**</property>
        
    </provider>

</identityProviders>

authorizations.xml

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizations>
    <policies>
        <policy identifier="627410be-1717-35b4-a06f-e9362b89e0b7" resource="/tenants" action="R">
            <user identifier="d9e3d4d3-e7d2-3c6e-9a70-2602c3265b53"/>
        </policy>
        <policy identifier="15e4e0bd-cb28-34fd-8587-f8d15162cba5" resource="/tenants" action="W">
            <user identifier="d9e3d4d3-e7d2-3c6e-9a70-2602c3265b53"/>
        </policy>
        <policy identifier="2dbc92a2-b091-3616-8e88-5078b9103b04" resource="/tenants" action="D">
            <user identifier="d9e3d4d3-e7d2-3c6e-9a70-2602c3265b53"/>
        </policy>
        <policy identifier="ff96062a-fa99-36dc-9942-0f6442ae7212" resource="/policies" action="R">
            <user identifier="d9e3d4d3-e7d2-3c6e-9a70-2602c3265b53"/>
        </policy>
        <policy identifier="ad99ea98-3af6-3561-ae27-5bf09e1d969d" resource="/policies" action="W">
            <user identifier="d9e3d4d3-e7d2-3c6e-9a70-2602c3265b53"/>
        </policy>
        <policy identifier="01b87cb5-c0b6-342d-b108-d8bc03ab5cde" resource="/policies" action="D">
            <user identifier="d9e3d4d3-e7d2-3c6e-9a70-2602c3265b53"/>
        </policy>
        <policy identifier="9d182b11-ebe3-3a7a-8731-98ce6d6e44fd" resource="/buckets" action="R">
            <user identifier="d9e3d4d3-e7d2-3c6e-9a70-2602c3265b53"/>
        </policy>
        <policy identifier="dfbf3c51-fdec-3328-b169-3b54eb033147" resource="/buckets" action="W">
            <user identifier="d9e3d4d3-e7d2-3c6e-9a70-2602c3265b53"/>
        </policy>
        <policy identifier="05b96464-9ec8-312a-8459-67812a8b48c1" resource="/buckets" action="D">
            <user identifier="d9e3d4d3-e7d2-3c6e-9a70-2602c3265b53"/>
        </policy>
        <policy identifier="2fd3fcf5-b10f-33fa-8d8e-b262fa34815e" resource="/actuator" action="R">
            <user identifier="d9e3d4d3-e7d2-3c6e-9a70-2602c3265b53"/>
        </policy>
        <policy identifier="2f470357-e82c-38ee-8062-ab6388d6ec75" resource="/actuator" action="W">
            <user identifier="d9e3d4d3-e7d2-3c6e-9a70-2602c3265b53"/>
        </policy>
        <policy identifier="3ee4703f-94ca-33c2-8060-17f5d313f560" resource="/actuator" action="D">
            <user identifier="d9e3d4d3-e7d2-3c6e-9a70-2602c3265b53"/>
        </policy>
        <policy identifier="0eaa47b9-e409-304e-8682-30d1b0d86d05" resource="/swagger" action="R">
            <user identifier="d9e3d4d3-e7d2-3c6e-9a70-2602c3265b53"/>
        </policy>
        <policy identifier="cf4d8390-5ac7-3ff0-82ce-a274b5f88b21" resource="/swagger" action="W">
            <user identifier="d9e3d4d3-e7d2-3c6e-9a70-2602c3265b53"/>
        </policy>
        <policy identifier="ac587f43-6e1c-3890-81fd-83b4df2e678e" resource="/swagger" action="D">
            <user identifier="d9e3d4d3-e7d2-3c6e-9a70-2602c3265b53"/>
        </policy>
        <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270" resource="/proxy" action="W">
            <user identifier="d9e3d4d3-e7d2-3c6e-9a70-2602c3265b53"/>
        </policy>
    </policies>
</authorizations>

users.xml

cat users.xml 
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<tenants>
    <groups/>
    <users>
        <user identifier="d9e3d4d3-e7d2-3c6e-9a70-2602c3265b53" identity="****"/>
    </users>
</tenants>

could you please point me what i missed?

Thanks,

Paul

[#4 · Accepted solution]

@Paul Yang 

A couple observation based on provided information:

1.  You have need clientAuth enabled

nifi.registry.security.needClientAuth=true

 With this enabled in NiFi-Registry the only authentication method supported will be 2-way TLS.  If you want to support other authentication methods lik Spnego, LDAP, and/or kerberos, this property must be false.

2. Your users.xml (used for user authorization and not authentication) only contains one user.  I am assuming the user is : 

CN=arch-fndtf04.beta1.fn, OU=NIFI

And your authorizations.xml also show that this is the only user that has been authorized to a bunch of policies.  Your ldap user you want to authorize must exist in the users.xml file.  My guess here is that you did not set the "Initial admin" user in your authorizers.xml (which you did not share).  There are two possible providers in the authorizers were you can set the initial admin (file-user-group-provider - Only set here if your initial admin user is NOT coming from the ldap-user-group-provider.  file-access-provider - initial admin must be set here so initial admin policies are created for this user.

3. The users.xml and authorizations.xml files are only generated if they do not already exist.  If you go back and add an initial admin to your authorizers.xml, you will need to delete/rename the existing users.xml and authorizations.xml files o new ones can be generated in startup.

4. Since need clientAuth was set to true and UI clearly shows that user string: 

CN=arch-fndtf04.beta1.fn, OU=NIFI

successfully authenticated, your browser must have this client certificate loaded and presented to the server when you navigated to the URL for your NiFi-Registry.  I really see no reason why this certificate was loaded int to your browser.  NiFi-Registry, even when need clientAuth is false, will always try mutual TLS authentication first (becomes a WANT instead of REQUIRE when need clientAuth is false).  If no client certificate is provided in TLS handshake, next auth method tried is Spnego (if configured), and finally a configured login provider from the identity-providers.xml.

Hope this info helps with correcting your setup issues.
Thanks,

Matt

[#5]

@MattWho 

Thanks for your detail answers. I almost to get win.

Unfortunately, I can not get sync policy between NIFI and NIFI Registry with my ldap account. I must to config my node identity as a user like CN=arch-fndtf04.beta1.fn, OU=NIFI  and grant it proxy access policy, so can import the bucket and commit the version. If i remove the user CN=arch-fndtf04.beta1.fn, OU=NIFI or i remove the proxy access policy of it in NIFI Registry. The NIFI GUI will show the "?" on top left corner of process group picture.

Could you help me how to avoid the issue?

Paul

[#6 · Accepted solution]

@Paul Yang 

I was in no way implying that you should have removed your NiFi nodes DN as a user identity in the NiFi-Registry.   The DN for every NiFi node must exist in NiFi-Registry and have been granted both proxy and read on "Can Manage Buckets" policies.

NiFi nodes will regularly read the buckets in the NiFi-Registry to see if a newer version of your Version controlled PG exists (this is why read  on "Can manage buckets" is needed.).  The "?" is displayed when the NiFi nodes can not read the bucket.

When a user in NiFi performs a version control action, the node will proxy the request n behalf of that user to the NiFi-Registry.  This is why all NiFi nodes must exist as users in NiFi-Registry and have the proxy policy granted to them.

Only your initial admin user should have all policies except proxy.  That user should never be proxying anything.

Thanks,
Matt

[#7]

@MattWho 

Follow your points i got win.

Thank you a lot.

Paul