Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Will enabling HTTPS and disabling HTTP on a single node in my cluster break connection with the rest of the cluster?

avatar
Explorer

I'm attempting to enable authentication on my NiFi server that is in a cluster with several other hosts. If I enabled HTTPS for webUI authentication, without changing any of the other hosts on the cluster, will this cause any complications with their communication with one another?

1 ACCEPTED SOLUTION

avatar
Super Mentor

@Cl0ck 

 

The basics:
1. NiFi must be secured before any form of Authentication and authorization can be utilized
2. You can not have a mix of secured and unsecured nodes in a single NiFi cluster.  Unsecured NiFi nodes would not be able to communicate with the secured NiFi nodes.
3. Securing NiFi requires that you provide each NiFi node with a keystore file that contains a single PrivateKeyEntry that includes at a minimum the following:
--- ExtendedKeyUsage supporting both clientAuth and serverAuth
--- SubjectAlternativeName(s) that match the exact hostname for the NiFi node that keystore is being installed on.
--- The keystore and key passwords need to be the same.
4. Securing NiFi also requires a truststore file which includes all the trust authorities.  These TrustedCertEntries can be the public certs of the root and intermediate Certificate Authorities (CAs) or the public cert for any self signed certs you may have created.

 

Note: The NiFi CA makes it easy to setup a CA and sign certificates for your nodes; however, it is not a full featured CA and is not recommended for production use.

Securing NiFi requires the setting of the following properties in the nifi.properties file:

nifi.security.keyPasswd=
nifi.security.keystore=/<path to>/keystore.jks
nifi.security.keystorePasswd=
nifi.security.keystoreType=JKS
nifi.security.truststore=/<path to>/truststore.jks
nifi.security.truststorePasswd=
nifi.security.truststoreType=JKS

nifi.web.https.host=<hostname>
nifi.web.https.network.interface.default=
nifi.web.https.port=<secure port>

nifi.cluster.protocol.is.secure=True

nifi.security.user.authorizer=

(optional) nifi.security.user.login.identity.provider=


Once NiFi is configured to be secure, you need to consider how your users with authenticate:
By default once secured, NiFi will require that all users authenticate via client/user TLS certificates.
However, NiFi offers a variety of additional authentication methods that can be configured as additions to the TLS authentication.
1. TLS authentication (always attempted first)
2. Spnego (configured in nifi.properties and attempted second if configured)
3. Login provider (configured in the login-identity-providers.xml.  Option include LDAP, kerberos, knox, and OpenID connect. Can only configure one and this is attempted third if neither one or two resulted in a client authenctication)

You mentioned that you want to use LDAP, so you would need to configure the ldap-provider in the login-identity-providers.xml and set the property "nifi.security.user.login.identity.provider=ldap-provider" in the nifi.properties file.

 

After authentication comes authorization... This is what the now authenticated user is allowed to do/access within your NiFi.  Authorization configuration is done via the authorizers.xml file.

It is easiest to read this file from the bottom up.
at the bottom you should have an "authorizer":

<authorizer>
            <identifier>managed-authorizer</identifier>
            <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
            <property name="Access Policy Provider">file-access-policy-provider</property>
    </authorizer>

Above is for the managed-authorizer which you would then reference in the nifi.properties file using the property "nifi.security.user.authorizer=managed-authorizer"

 

You can see this authorizer calls a "file-access-policy-provider" which you must find above this entry in the authorizers.xml:

    <accessPolicyProvider>
        <identifier>file-access-policy-provider</identifier>
        <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
        <property name="User Group Provider">composite-user-group-provider</property>
        <property name="Node Group"></property>
        <property name="Initial Admin Identity"><username of ldap user who will act as the initial admin user></property>
        <property name="Authorizations File">/<path to>/authorizations.xml</property>
        <property name="Node Identity 2"><dn of node 1 in cluster></property>
        <property name="Node Identity 3"><dn of node 2 in cluster></property>
        <property name="Node Identity 4"><dn of node N in cluster></property>
    </accessPolicyProvider>

The above is responsible for setting up some initial necessary authorizations for your initial admin user and NiFi nodes in your cluster.  You can see this provider calls another provider "composite-user-group-provider" which you will find further up in the authorizers.xml.  This provider builds the authorizations.xml file.

 

Note: Authorizations.xml file is only created if it does not already exist.  If it exists already, no changes will be made to it if you modify this file.  Expectation is that all new authorizations are made via the NiFi UI by your initial admin user.

The next three providers control where the NiFi authorizer learns about the users to which authorizations will be granted.

    <userGroupProvider>
        <identifier>composite-user-group-provider</identifier>
     <class>org.apache.nifi.authorization.CompositeConfigurableUserGroupProvider</class>
        <property name="User Group Provider 1">ldap-user-group-provider</property>
        <property name="Configurable User Group Provider">file-user-group-provider</property>
    </userGroupProvider>

    <userGroupProvider>
        <identifier>file-user-group-provider</identifier>
        <class>org.apache.nifi.authorization.FileUserGroupProvider</class>
        <property name="Users File">/<path to>/users.xml</property>
        <property name="Initial User Identity 2"><dn of NiFi node 1></property>
        <property name="Initial User Identity 3"><dn of NiFi node 2></property>
        <property name="Initial User Identity 4"><dn of NiFi node 3></property>
    </userGroupProvider>

    <userGroupProvider>
        <identifier>ldap-user-group-provider</identifier>
        <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
        <property name="Authentication Strategy">SIMPLE</property>
        <property name="Manager DN"><dn of ldap manger></property>
        <property name="Manager Password"><ldap manager password></property>
        <property name="TLS - Keystore"></property>
        <property name="TLS - Keystore Password"></property>
        <property name="TLS - Keystore Type"></property>
        <property name="TLS - Truststore"></property>
        <property name="TLS - Truststore Password"></property>
        <property name="TLS - Truststore Type"></property>
        <property name="TLS - Client Auth"></property>
        <property name="TLS - Protocol"></property>
        <property name="TLS - Shutdown Gracefully"></property>
        <property name="Referral Strategy">FOLLOW</property>
        <property name="Connect Timeout">10 secs</property>
        <property name="Read Timeout">10 secs</property>
        <property name="Url">ldap://<hosntame>:<port></property>
        <property name="Page Size">500</property>
        <property name="Sync Interval">30 mins</property>

        <property name="User Search Base"></property>
        <property name="User Object Class"></property>
        <property name="User Search Scope">SUBTREE</property>
        <property name="User Search Filter"></property>
        <property name="User Identity Attribute"></property>
        <property name="User Group Name Attribute"></property>
        <property name="User Group Name Attribute - Referenced Group Attribute"></property>

        <property name="Group Search Base"></property>
        <property name="Group Object Class"></property>
        <property name="Group Search Scope">SUBTREE</property>
        <property name="Group Search Filter"></property>
        <property name="Group Name Attribute"></property>
        <property name="Group Member Attribute"></property>
        <property name="Group Member Attribute - Referenced User Attribute"></property>
    </userGroupProvider>

Of course you will need to fill in all the required ldap properties to sync you users and groups from your LDAP. 

The file-user-group-provider is used for creating any local users.  You will see above it created your NiFi nodes as local users in a users.xml file since it is very unlikely these NiFi servers will exist in ldap.

 

Note; The users.xml is only generated once.  If it already exists it will not be updated with any future changes made to this file.  Expectations are that additional local users are added by your initial admin manually via the NiFi UI.

 

Note: Always use search filters to limit number of users and groups returned to only those that will be accessing your NiFi.  NiFi holds all these returned users/groups in heap memory, so you want to avoid syncing the entire ldap.

 

Other things to keep in mind...
1. NiFi is case sensitive.  User "John Smith" is not the same as user "john smith".

2. The user string that results from successful authentication must match EXACTLY" with the user string returned by the ldap-user-group-provider or file-user-group-provider.

NIFi identity mapping patterns can be used to trim/modify the strings returned by both the authentication provider and authorization providers so they match:

examples:

nifi.security.group.mapping.pattern.anygroup=(.*?)
nifi.security.group.mapping.value.anygroup=$1
nifi.security.group.mapping.transform.anygroup=LOWER

nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?)$
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=LOWER

nifi.security.identity.mapping.pattern.kerb=^(.*?)@(.*?)$
nifi.security.identity.mapping.value.kerb=$1
nifi.security.identity.mapping.transform.kerb=LOWER

 

Hope this helps you get started,

Matt

 

View solution in original post

9 REPLIES 9

avatar
Explorer

I've been trying to get this to work for many days now and keep running into issues, so I would appreciate any input anyone has to offer on this. I need to enable a self-signed TLS 1.2 cert for HTTPS on the single NiFi node in my cluster in order to authenticate via LDAP.

Jetty is the current webserver running on NiFi, and everything is working fine except there is no authentication method so anyone who has access to it can go in and make changes. I've read the documentation provided quite a bit, but still have many questions unanswered.

Is NiFi Toolkit CA  primarily for securing multiple NiFi Nodes, or should I be using it to get HTTPS set up on the server?

avatar
New Contributor

Hi buddy, exactly trying to do kind of same. Did you get any way out yet? If yes would you mind sharing?

 

Regards,

Sloth Smith

avatar
Explorer

Any luck getting it working? I'm still working on it to this day, but I think I'm gonna finish it tonight. If so, I can help you when I know my method works.

avatar
Super Mentor

@Cl0ck 

 

The basics:
1. NiFi must be secured before any form of Authentication and authorization can be utilized
2. You can not have a mix of secured and unsecured nodes in a single NiFi cluster.  Unsecured NiFi nodes would not be able to communicate with the secured NiFi nodes.
3. Securing NiFi requires that you provide each NiFi node with a keystore file that contains a single PrivateKeyEntry that includes at a minimum the following:
--- ExtendedKeyUsage supporting both clientAuth and serverAuth
--- SubjectAlternativeName(s) that match the exact hostname for the NiFi node that keystore is being installed on.
--- The keystore and key passwords need to be the same.
4. Securing NiFi also requires a truststore file which includes all the trust authorities.  These TrustedCertEntries can be the public certs of the root and intermediate Certificate Authorities (CAs) or the public cert for any self signed certs you may have created.

 

Note: The NiFi CA makes it easy to setup a CA and sign certificates for your nodes; however, it is not a full featured CA and is not recommended for production use.

Securing NiFi requires the setting of the following properties in the nifi.properties file:

nifi.security.keyPasswd=
nifi.security.keystore=/<path to>/keystore.jks
nifi.security.keystorePasswd=
nifi.security.keystoreType=JKS
nifi.security.truststore=/<path to>/truststore.jks
nifi.security.truststorePasswd=
nifi.security.truststoreType=JKS

nifi.web.https.host=<hostname>
nifi.web.https.network.interface.default=
nifi.web.https.port=<secure port>

nifi.cluster.protocol.is.secure=True

nifi.security.user.authorizer=

(optional) nifi.security.user.login.identity.provider=


Once NiFi is configured to be secure, you need to consider how your users with authenticate:
By default once secured, NiFi will require that all users authenticate via client/user TLS certificates.
However, NiFi offers a variety of additional authentication methods that can be configured as additions to the TLS authentication.
1. TLS authentication (always attempted first)
2. Spnego (configured in nifi.properties and attempted second if configured)
3. Login provider (configured in the login-identity-providers.xml.  Option include LDAP, kerberos, knox, and OpenID connect. Can only configure one and this is attempted third if neither one or two resulted in a client authenctication)

You mentioned that you want to use LDAP, so you would need to configure the ldap-provider in the login-identity-providers.xml and set the property "nifi.security.user.login.identity.provider=ldap-provider" in the nifi.properties file.

 

After authentication comes authorization... This is what the now authenticated user is allowed to do/access within your NiFi.  Authorization configuration is done via the authorizers.xml file.

It is easiest to read this file from the bottom up.
at the bottom you should have an "authorizer":

<authorizer>
            <identifier>managed-authorizer</identifier>
            <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
            <property name="Access Policy Provider">file-access-policy-provider</property>
    </authorizer>

Above is for the managed-authorizer which you would then reference in the nifi.properties file using the property "nifi.security.user.authorizer=managed-authorizer"

 

You can see this authorizer calls a "file-access-policy-provider" which you must find above this entry in the authorizers.xml:

    <accessPolicyProvider>
        <identifier>file-access-policy-provider</identifier>
        <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
        <property name="User Group Provider">composite-user-group-provider</property>
        <property name="Node Group"></property>
        <property name="Initial Admin Identity"><username of ldap user who will act as the initial admin user></property>
        <property name="Authorizations File">/<path to>/authorizations.xml</property>
        <property name="Node Identity 2"><dn of node 1 in cluster></property>
        <property name="Node Identity 3"><dn of node 2 in cluster></property>
        <property name="Node Identity 4"><dn of node N in cluster></property>
    </accessPolicyProvider>

The above is responsible for setting up some initial necessary authorizations for your initial admin user and NiFi nodes in your cluster.  You can see this provider calls another provider "composite-user-group-provider" which you will find further up in the authorizers.xml.  This provider builds the authorizations.xml file.

 

Note: Authorizations.xml file is only created if it does not already exist.  If it exists already, no changes will be made to it if you modify this file.  Expectation is that all new authorizations are made via the NiFi UI by your initial admin user.

The next three providers control where the NiFi authorizer learns about the users to which authorizations will be granted.

    <userGroupProvider>
        <identifier>composite-user-group-provider</identifier>
     <class>org.apache.nifi.authorization.CompositeConfigurableUserGroupProvider</class>
        <property name="User Group Provider 1">ldap-user-group-provider</property>
        <property name="Configurable User Group Provider">file-user-group-provider</property>
    </userGroupProvider>

    <userGroupProvider>
        <identifier>file-user-group-provider</identifier>
        <class>org.apache.nifi.authorization.FileUserGroupProvider</class>
        <property name="Users File">/<path to>/users.xml</property>
        <property name="Initial User Identity 2"><dn of NiFi node 1></property>
        <property name="Initial User Identity 3"><dn of NiFi node 2></property>
        <property name="Initial User Identity 4"><dn of NiFi node 3></property>
    </userGroupProvider>

    <userGroupProvider>
        <identifier>ldap-user-group-provider</identifier>
        <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
        <property name="Authentication Strategy">SIMPLE</property>
        <property name="Manager DN"><dn of ldap manger></property>
        <property name="Manager Password"><ldap manager password></property>
        <property name="TLS - Keystore"></property>
        <property name="TLS - Keystore Password"></property>
        <property name="TLS - Keystore Type"></property>
        <property name="TLS - Truststore"></property>
        <property name="TLS - Truststore Password"></property>
        <property name="TLS - Truststore Type"></property>
        <property name="TLS - Client Auth"></property>
        <property name="TLS - Protocol"></property>
        <property name="TLS - Shutdown Gracefully"></property>
        <property name="Referral Strategy">FOLLOW</property>
        <property name="Connect Timeout">10 secs</property>
        <property name="Read Timeout">10 secs</property>
        <property name="Url">ldap://<hosntame>:<port></property>
        <property name="Page Size">500</property>
        <property name="Sync Interval">30 mins</property>

        <property name="User Search Base"></property>
        <property name="User Object Class"></property>
        <property name="User Search Scope">SUBTREE</property>
        <property name="User Search Filter"></property>
        <property name="User Identity Attribute"></property>
        <property name="User Group Name Attribute"></property>
        <property name="User Group Name Attribute - Referenced Group Attribute"></property>

        <property name="Group Search Base"></property>
        <property name="Group Object Class"></property>
        <property name="Group Search Scope">SUBTREE</property>
        <property name="Group Search Filter"></property>
        <property name="Group Name Attribute"></property>
        <property name="Group Member Attribute"></property>
        <property name="Group Member Attribute - Referenced User Attribute"></property>
    </userGroupProvider>

Of course you will need to fill in all the required ldap properties to sync you users and groups from your LDAP. 

The file-user-group-provider is used for creating any local users.  You will see above it created your NiFi nodes as local users in a users.xml file since it is very unlikely these NiFi servers will exist in ldap.

 

Note; The users.xml is only generated once.  If it already exists it will not be updated with any future changes made to this file.  Expectations are that additional local users are added by your initial admin manually via the NiFi UI.

 

Note: Always use search filters to limit number of users and groups returned to only those that will be accessing your NiFi.  NiFi holds all these returned users/groups in heap memory, so you want to avoid syncing the entire ldap.

 

Other things to keep in mind...
1. NiFi is case sensitive.  User "John Smith" is not the same as user "john smith".

2. The user string that results from successful authentication must match EXACTLY" with the user string returned by the ldap-user-group-provider or file-user-group-provider.

NIFi identity mapping patterns can be used to trim/modify the strings returned by both the authentication provider and authorization providers so they match:

examples:

nifi.security.group.mapping.pattern.anygroup=(.*?)
nifi.security.group.mapping.value.anygroup=$1
nifi.security.group.mapping.transform.anygroup=LOWER

nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?)$
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=LOWER

nifi.security.identity.mapping.pattern.kerb=^(.*?)@(.*?)$
nifi.security.identity.mapping.value.kerb=$1
nifi.security.identity.mapping.transform.kerb=LOWER

 

Hope this helps you get started,

Matt

 

avatar
Explorer

Thanks so much Matt!! So, security is working now. BUT no luck logging in with ldap yet. Do I need to configure the safety valves in order to access

 

login-identity-providers.xml
[root@nifi /]# find . -name login-identity-providers.xml
./run/cloudera-scm-agent/process/196-nifi-NIFI_NODE/login-identity-providers.xml
./run/cloudera-scm-agent/process/196-nifi-NIFI_NODE/aux/defaults/login-identity-providers.xml
./run/cloudera-scm-agent/process/195-nifi-NIFI_NODE/login-identity-providers.xml
./run/cloudera-scm-agent/process/195-nifi-NIFI_NODE/aux/defaults/login-identity-providers.xml
./run/cloudera-scm-agent/process/194-nifi-NIFI_NODE/login-identity-providers.xml
./run/cloudera-scm-agent/process/194-nifi-NIFI_NODE/aux/defaults/login-identity-providers.xml
./run/cloudera-scm-agent/process/181-nifi-NIFI_NODE/login-identity-providers.xml
./run/cloudera-scm-agent/process/181-nifi-NIFI_NODE/aux/defaults/login-identity-providers.xml

 
Are there multiple processes running as a security measure, or is something on configured properly? I have no idea which login-identity-providers.xml to edit!

avatar
Super Mentor

@Cl0ck 

 

You did not mention you were using CM.  Every time a configuration change is made and NiFi service is restarted via CM, a new configuration folder is created.  The most recently created is what is being used.  From what you shared here, the configs inside 

196-nifi-NIFI_NODE

are being used.

You cannot hand edit these config files on disk when using CM.  All configuration must be done in CM.

Those properties/configs which are not exposed would require safety valves to setup.

 

If you have a support contract, i recommend opening a support case if you need help walking through this setup.

 

Thanks,

Matt

avatar
Explorer

I assumed as such, actually! My bad for not communicating that I'm configuring from within CM. We don't have a support license yet, but are hoping to ASAP.

Just for basic, SIMPLE LDAP authentication should I need to configure safety valves? It seems like I should be able to get it working with the configurations available. However, it's getting stuck somewhere.

I can connect to the server via HTTPS and I get a login screen. Should I be able to log in without LDAP using the initial admin + master password?

avatar
Super Mentor

@Cl0ck 

 

NiFi has no local users accounts which can be used for authentication.

Your initial admin user should have been one of your LDAP users which would require you to provide that ldap username and ldap user's password at the login prompt displayed by NiFi.

The Initial Admin property used by NiFi is only for establishing the initial set of authorization policies needed to allow that user to be the admin user.  It has nothing to do with user authentication at all.  Only used post successful authentication for authorization purposes.

CM allows you to setup the login-Identity-providers.xml file without needing to create any safety valves.  Those properties are all exposed by default in the available configs.  You can filter  by the word "login" to expose all those properties.

Safety valves are needed if you want to setup NiFi to sync users and groups from ldap in to NiFi's authorizer.  You can optionally just use the default file based authorization which would then require your initial admin user to login to the UI and add any additional users and associate authorization policies to them.  With the ldap-user-group-provider configured NiFi will sync users and groups from ldap so the the initial admin user would only need to associate policies to those ldap users (no need to also add the users and/or groups).

Hope this clear some things up here.
Matt

avatar
Explorer

Thanks so much Matt!! You've been a huge help in getting my mind wrapped around all of this. If you can't tell, I'm a bit new to authentication and authorization! I really appreciate everything you've done to point me in the right direction. If things go as planned, I *should* have a NiFi node working by the end of the day!

Here's one more for you, though. I'm having to reinstall the NiFi service on a new host because some property values are messed up after all of my tinkering trying to get things to work. The node will start up, HTTPS will be working, I can successfully log into the WebUI, but then after ~5 minutes or so, something happens and it reverts to trying to use HTTP and reports that it is trying to connect to the site on the HTTP port and fails to do so. I believe everything is configured in CM properly but there are some local configs that aren't right, or profiles that need to be deleted in order for new (correct) profiles to be created automatically.

If this rings any bells I would love to learn more about how to fix it, but for now it seems the best thing to do is to do a fresh install. Aloha!