Unable to open NIFI web UI after TLS

[#1]

Hi All,

I enabled TLS for NIFI web UI ( CDF ) , while services are running fine on cluster I'm unable to access NIFI web UI from my browser. Below are the steps I followed please suggest what might be causing issue ?

I repeated below steps for all the machines in my nifi cluster

1. Received signed host certificate from IT team ( <hostname>.pem ) , also rootca (root.pem)

2. Copy the JDK cacerts file to jssecacerts as follows:

sudo cp $JAVA_HOME/jre/lib/security/cacerts $JAVA_HOME/jre/lib/security/jssecacerts

3. import rootca cert into JKS store

sudo $JAVA_HOME/bin/keytool -importcert -alias rootca -keystore $JAVA_HOME/jre/lib/security/jssecacerts -file /opt/cloudera/security/pki/root.pem

4. Created JKS and imported host certificate in keystore.

$JAVA_HOME/bin/keytool -genkeypair -alias $(hostname -f) -keyalg RSA -keystore /opt/cloudera/security/pki/$(hostname -f).jks -keysize 2048 -dname "CN=$(hostname -f),OU=Engineering,O=Cloudera,L=Singapore,ST=Singapore,C=Singapore" -ext san=dns:$(hostname -f)

sudo $JAVA_HOME/bin/keytool -importcert -alias $(hostname -f) -file /opt/cloudera/security/pki/$(hostname -f).pem -keystore /opt/cloudera/security/pki/$(hostname -f).jks

5. creating symlinks

sudo ln -s /opt/cloudera/security/pki/$(hostname -f).pem /opt/cloudera/security/pki/agent.pem

sudo ln -s /opt/cloudera/security/pki/$(hostname -f).jks /opt/cloudera/security/pki/server.jks

6. Enabled TLS from Cloudera Manager for NIFI

7. Restarted services from Cloudera manager 

8. Unable to access from Browser 

[#2 · Accepted solution]

@chhaya_vishwaka 

NiFi keystore requirements:
1. NiFi Keystore must contain only ONE PrivateKeyEntry
2. NiFi does not support using wildcards in certificate DNs
3. PrivateKeyEntry must support both clientAuth and serverAuth Extended Key Usage (EKU)
4. PrivateKeyEntry must contain at least one Subject Alternative Name (SAN) entry that matches the hostname on which the keystore is being used.

You can obtain a verbose output of your keystore using the keytool command to verify all of the above criteria are met:

keytool -v -list -keystore <your keystore file>

Once above is verified, you need to make sure the truststore being used by all your NiFi nodes contains the complete certificate trust chain for your server certificates. This means if your server certificate was signed by an intermediate CA, your truststore must contain the public certificate for that intermediate CA as well as the public certificate for the signer of the intermediate CA.  Yo u may have several intermediate CAs in-line before you finally reach the rootCA (owner and issuer are the same).

Once above is verified, now it should be just a matter of authenticated and authorized access to your secured NiFi.  By default when you enable TLS in NiFi, all clients/users are expected to authenticate with NiFi using a user certificate which they load in their browser.  NiFi does not have local user for purpose of authentication.

NiFi can be configured to support additional forms of user authentication such as Spnego, LDAP, kerberos, OpenID connect, etc.  Refer to the admin guide and user guide for more detail.  Your browser screenshot indicates that you do not have a client/user certificates loaded in your browser which your secured NiFi can trust and since NiFi did not redirect you to login page, you do not have a login provider configured in your login-identity-providers.xml (or nifi.properties file is not configured to use it if you do).

Hope this detail helps you on the path to resolving your issue,

Matt

[#3 · Accepted solution]

@chhaya_vishwaka 

"Insufficient Permissions" indicates the issue is with user/client authorization and not authentication anymore.

Note: you may or may not see the "log out" option.

You should tail the nifi-user.log and then try accessing the NiFi UI to get the full log output for the user action that was attempted.

For example:

2020-02-07 15:32:51,925 INFO [NiFi Web Server-26362] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[nifiuser1], groups[nifiDFMeast, users] does not have permission to access the requested resource. Unable to view the user interface. Returning Forbidden response.
2020-02-07 15:32:51,927 DEBUG [NiFi Web Server-26362] o.a.n.w.a.c.AccessDeniedExceptionMapper
org.apache.nifi.authorization.AccessDeniedException: Unable to view the user interface.

If you are not ever seeing the canvas, your authenticated user does not have the "view the user interface" (/flow policy) granted to your authenticated user identity string.  As you can see from above nifi-user.log output, we can confirm this.

What we also learn from this log output is that the user who successfully authenticated is being identified with user string "nifiuser1" and is a member of two groups "nifiDFMeast" and "users".

NiFi authorization is handled by which ever authorizer you have configured in your NiFi authorixzers.xml and referenced my the following property in your nifi.properties file:

nifi.security.user.authorizer=managed-authorizer

By default a secured NiFi will use a file based authorizer which relies on a "users.xml" (associates locally created known users to locally created known groups) and an "authorizations.xml" (associates the UUIDs assigned to the users and groups to various authorization policies.

So using example above, my users.xml file would need to contain user "nifiuser1" and groups "nifiDFMeast" and "users" (all case sensitive).  Each of those strings would be assigned a UUID.  In order for my user to see the NiFi canvas, one of those assigned UUIDs would need to be assigned to the /flow policy in the authorizations.xml file.

NiFi on first startup only creates the users.xml and authorizations.xml file based on initial admin identity and node Identity values configured in the authorizers.xml file.  If you have already started NiFi secure and later edit these identities, those changes will not be reflected in either file.   Remove these two files an restart NiFi so they are created new again with your changes (this should only be done during initial setup of a secure NiFi).  Once your initial admin user is able to access the UI, all additional users and authorizations policies would be added via the NiFi UI.

Hope this helps,

Matt

[#4 · Accepted solution]

@chhaya_vishwaka 

You can see from the nifi-user.log that the DN passed to NiFi was:

CN=nifi-host1, OU=nifi, O=nifi, L=SG, ST=SG, C=SG

Where did cert with above DN come from?
Is a cert with this DN loaded on your browser?

The NiFi user log goes on to tell you that the above authenticated string was not authorized for "view the user interface":

[CN=nifi-host1, OU=nifi, O=nifi, L=SG, ST=SG, C=SG], groups[] does not have permission to access the requested resource. Unable to view the user interface. Returning Forbidden response.

This certificate does not have "Admin" anywhere in its DN.  Did you maybe load your NiFi node certificates in to your browser also?  When your browser prompted you to select a certificate to use, did you maybe select the wrong one?

I also see from above that the entire DN is being passed for authorization which tells me you do not have an identity mapping pattern setup in your nifi.properties file to trim a portion of the DN (CN value most commonly) from the full DN.    If you used "Admin" as your initial admin identity, then the resulting string after authentication and post any configured identity mapping is applied must match exactly to "Admin".

https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#identity-mapping-properties

As an example, the following identity mapping pattern properties would result in the above DN being trimmed so that only "nifi-host1" would be passed to the authorizer.

nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=nifi, O=nifi, L=SG, ST=SG, C=SG$
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=NONE

Note:  The transform property is used to normalize the resulting mapped value to all UPPER or all LOWER case if needed.  NONE results in no transform.
Also keep in mind that mapping patterns will also be applied to your server certificates DNs.  So if your server certificate DN matches the pattern regex also, only the server certs CN value will be passed to authorizer. So if you use identity patterns, you are likely going to need to update your "Initial User Identity <num>" and "Node Identity <num>" in your authorizers.xml file to use the CN value instead of the full DN for each of your nodes.

Hope this helps,

Matt

[#5 · Accepted solution]

@chhaya_vishwaka 

NiFi keystore requirements:
1. NiFi Keystore must contain only ONE PrivateKeyEntry
2. NiFi does not support using wildcards in certificate DNs
3. PrivateKeyEntry must support both clientAuth and serverAuth Extended Key Usage (EKU)
4. PrivateKeyEntry must contain at least one Subject Alternative Name (SAN) entry that matches the hostname on which the keystore is being used.

You can obtain a verbose output of your keystore using the keytool command to verify all of the above criteria are met:

keytool -v -list -keystore <your keystore file>

Once above is verified, you need to make sure the truststore being used by all your NiFi nodes contains the complete certificate trust chain for your server certificates. This means if your server certificate was signed by an intermediate CA, your truststore must contain the public certificate for that intermediate CA as well as the public certificate for the signer of the intermediate CA.  Yo u may have several intermediate CAs in-line before you finally reach the rootCA (owner and issuer are the same).

Once above is verified, now it should be just a matter of authenticated and authorized access to your secured NiFi.  By default when you enable TLS in NiFi, all clients/users are expected to authenticate with NiFi using a user certificate which they load in their browser.  NiFi does not have local user for purpose of authentication.

NiFi can be configured to support additional forms of user authentication such as Spnego, LDAP, kerberos, OpenID connect, etc.  Refer to the admin guide and user guide for more detail.  Your browser screenshot indicates that you do not have a client/user certificates loaded in your browser which your secured NiFi can trust and since NiFi did not redirect you to login page, you do not have a login provider configured in your login-identity-providers.xml (or nifi.properties file is not configured to use it if you do).

Hope this detail helps you on the path to resolving your issue,

Matt

[#6]

Hi @MattWho  I have checked configurations as mentioned by you but still I'm stuck at same issue, can you advise ?

[#7]

I'm stuck user certificate step if someone can help to understand on how to get these certs 

my nifi UI give "Insufficient Permissions"

[#8 · Accepted solution]

@chhaya_vishwaka 

"Insufficient Permissions" indicates the issue is with user/client authorization and not authentication anymore.

Note: you may or may not see the "log out" option.

You should tail the nifi-user.log and then try accessing the NiFi UI to get the full log output for the user action that was attempted.

For example:

2020-02-07 15:32:51,925 INFO [NiFi Web Server-26362] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[nifiuser1], groups[nifiDFMeast, users] does not have permission to access the requested resource. Unable to view the user interface. Returning Forbidden response.
2020-02-07 15:32:51,927 DEBUG [NiFi Web Server-26362] o.a.n.w.a.c.AccessDeniedExceptionMapper
org.apache.nifi.authorization.AccessDeniedException: Unable to view the user interface.

If you are not ever seeing the canvas, your authenticated user does not have the "view the user interface" (/flow policy) granted to your authenticated user identity string.  As you can see from above nifi-user.log output, we can confirm this.

What we also learn from this log output is that the user who successfully authenticated is being identified with user string "nifiuser1" and is a member of two groups "nifiDFMeast" and "users".

NiFi authorization is handled by which ever authorizer you have configured in your NiFi authorixzers.xml and referenced my the following property in your nifi.properties file:

nifi.security.user.authorizer=managed-authorizer

By default a secured NiFi will use a file based authorizer which relies on a "users.xml" (associates locally created known users to locally created known groups) and an "authorizations.xml" (associates the UUIDs assigned to the users and groups to various authorization policies.

So using example above, my users.xml file would need to contain user "nifiuser1" and groups "nifiDFMeast" and "users" (all case sensitive).  Each of those strings would be assigned a UUID.  In order for my user to see the NiFi canvas, one of those assigned UUIDs would need to be assigned to the /flow policy in the authorizations.xml file.

NiFi on first startup only creates the users.xml and authorizations.xml file based on initial admin identity and node Identity values configured in the authorizers.xml file.  If you have already started NiFi secure and later edit these identities, those changes will not be reflected in either file.   Remove these two files an restart NiFi so they are created new again with your changes (this should only be done during initial setup of a secure NiFi).  Once your initial admin user is able to access the UI, all additional users and authorizations policies would be added via the NiFi UI.

Hope this helps,

Matt

[#9]

@MattWho  I added client certificate in my browser . Below are steps i followed

I setup initial admin indentity in nifi conf as "Admin"

Got client cert for "Admin" from IT team

added cert in my browser.

Below is logline from my nifi-user.log.

"2020-02-10 11:18:54,677 INFO [main] o.a.n.a.FileAccessPolicyProvider Populating authorizations for Initial Admin: Admin
2020-02-10 11:18:54,693 INFO [main] o.a.n.a.FileAccessPolicyProvider Authorizations file loaded at Mon Feb 10 11:18:54 SGT 2020
2020-02-10 11:19:12,609 INFO [NiFi Web Server-48] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi.. Returning Conflict response.
2020-02-10 11:19:12,641 INFO [NiFi Web Server-36] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: OpenId Connect is not configured.. Returning Conflict response."

2020-02-10 11:19:12,666 INFO [NiFi Web Server-48] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=nifi-host1, OU=nifi, O=Liongate, L=nifi, ST=SG, C=SG) GET https://localhost:8443/nifi-api/flow/current-user (source ip: <ip>)
2020-02-10 11:19:12,673 INFO [NiFi Web Server-48] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=nifi-host1, OU=nifi, O=nifi, L=SG, ST=SG, C=SG
2020-02-10 11:19:12,738 INFO [NiFi Web Server-48] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[CN=nifi-host1, OU=nifi, O=nifi, L=SG, ST=SG, C=SG], groups[] does not have permission to access the requested resource. Unable to view the user interface. Returning Forbidden response.
2020-02-10 11:22:11,245 INFO [main] o.a.n.a.FileUserGroupProvider Users/Groups file loaded at Mon Feb 10 11:22:11 SGT 2020

[#10 · Accepted solution]

@chhaya_vishwaka 

You can see from the nifi-user.log that the DN passed to NiFi was:

CN=nifi-host1, OU=nifi, O=nifi, L=SG, ST=SG, C=SG

Where did cert with above DN come from?
Is a cert with this DN loaded on your browser?

The NiFi user log goes on to tell you that the above authenticated string was not authorized for "view the user interface":

[CN=nifi-host1, OU=nifi, O=nifi, L=SG, ST=SG, C=SG], groups[] does not have permission to access the requested resource. Unable to view the user interface. Returning Forbidden response.

This certificate does not have "Admin" anywhere in its DN.  Did you maybe load your NiFi node certificates in to your browser also?  When your browser prompted you to select a certificate to use, did you maybe select the wrong one?

I also see from above that the entire DN is being passed for authorization which tells me you do not have an identity mapping pattern setup in your nifi.properties file to trim a portion of the DN (CN value most commonly) from the full DN.    If you used "Admin" as your initial admin identity, then the resulting string after authentication and post any configured identity mapping is applied must match exactly to "Admin".

https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#identity-mapping-properties

As an example, the following identity mapping pattern properties would result in the above DN being trimmed so that only "nifi-host1" would be passed to the authorizer.

nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=nifi, O=nifi, L=SG, ST=SG, C=SG$
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=NONE

Note:  The transform property is used to normalize the resulting mapped value to all UPPER or all LOWER case if needed.  NONE results in no transform.
Also keep in mind that mapping patterns will also be applied to your server certificates DNs.  So if your server certificate DN matches the pattern regex also, only the server certs CN value will be passed to authorizer. So if you use identity patterns, you are likely going to need to update your "Initial User Identity <num>" and "Node Identity <num>" in your authorizers.xml file to use the CN value instead of the full DN for each of your nodes.

Hope this helps,

Matt

[#11]

@MattWho 

thanks for the inputs I was able to resolve user certificate issue, but my UI is stuck with below issue. I'm not sure why its still referring to http port. It's kind of weird , can you please advise ?

java.net.ConnectException: Failed to connect to hostname/ip:8080

[#12 · Accepted solution]

@chhaya_vishwaka 

Please stop your NiFi nodes and delete the local NiFi "state" directory on each of them.  Then restart your NiFi nodes to see if this helps.  Removing the local state will clear out the retained node information which may still contain info about your nodes before they were secured.

Thanks,

Matt

[#13]

@MattWho ill try to delete local state directory and restart nodes , is there anything else I should look into ?