Member since
07-30-2019
3058
Posts
1526
Kudos Received
888
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
31 | 10-07-2024 06:12 AM | |
216 | 10-02-2024 09:52 AM | |
216 | 10-02-2024 09:39 AM | |
118 | 09-27-2024 10:22 AM | |
213 | 09-27-2024 08:45 AM |
10-07-2024
06:12 AM
@newbeede The only Authentication provider provided with NiFi that supports a single managed user with password is the Single User provider. Apache NiFi create this authentication provider so that NiFi out-of-the-box could run securely over HTTPS. It is intended for non clustered use to make it easy to evaluate the product. The users.xml is created by the file-user-group-provider inside the authorizers.xml. It's is for authorization use only and has nothing to do with user/client authentication. In order to setup various authorization policies through the file-access-policy provider (also found in the authorizers,.xml and generates an authorizations.xml file), the authorizer must be aware of the user/client identities for which policies will be granted. The file-user-group-provider is just one of the available providers that creates these user identities within NiFi for the sole purpose of policy assignment. The file-user-group-provider is used to generate the Initial Admin Identity and the Identities for each of your cluster nodes. The users.xml file it creates will only initially contain those identities. The file-user-group-provider will ONLY generate the users.xml on NiFi startup if it does NOT already exist. Once the NiFi UI is accessed by the initial admin identity, additional user identities and policies are setup from within the UI. Additional user identities and group identities added through the NiFi UI will be added to the users.xml. If you are using the Single-User authentication provider and single-user authorizer, then users and polices will not present in the UI when accessed. The File-access-policy provider sets up admin polices for the defined initial admin identity and sets up initial node required policies. The policies are needed for that initial admin to be authorized to create additional user identities and configure additional policy authorizations from with the NiFi UI. The file-access-policy provider will ONLY generate the authorizations.xml file on NiFi startup if it does NOT already exist. Additional policies setup from with the NiFi UI are added to the authorizations.xml file. So to be clear, NiFi does allow you to dynamically add/remove additional user identities and dynamically add/modify/remove authorizations from the NiFi UI without needing a restart. When it comes to handling user authentication, a secured NiFi will always support client Authentication as first authentication method attempted. This is the only method used for the NiFi nodes to communicate with one another. When additional authentication method are configured, NiFi will still WANT a client auth certificate in the TLS exchange; however if one is not provided, NiFi will try the next authentication method configured. The most commonly used additional authentication method used are the LDAP-provider and Kerberos-provider. The LDAP-provider allows you to authenticate via your own LDAP/AD. Sample: <provider>
<identifier>ldap-provider</identifier>
<class>org.apache.nifi.ldap.LdapProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN"></property> <-- MUST be configured
<property name="Manager Password"></property> <-- MUST be configured
<property name="TLS - Keystore"></property>
<property name="TLS - Keystore Password"></property>
<property name="TLS - Keystore Type"></property>
<property name="TLS - Truststore"></property>
<property name="TLS - Truststore Password"></property>
<property name="TLS - Truststore Type"></property>
<property name="TLS - Client Auth"></property>
<property name="TLS - Protocol"></property>
<property name="TLS - Shutdown Gracefully"></property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://<hostname>:<port></property> <-- MUST be configured
<property name="User Search Base">CN=Users,DC=example,DC=com)</property> <-- MUST be configured
<property name="User Search Filter">cn={0}</property> <-- MUST be configured
<property name="Identity Strategy">USE_USERNAME</property> <-- MUST be configured
<property name="Authentication Expiration">12 hours</property>
</provider> USE_USERNAME: Upon successful LDAP/AD authentication, the user identity string as typed (case sensitive) in the username login window will be passed to the authorizer to look up policies assigned to the user identity. USE_DN: Will pass the DN returned from LDAP/AD after successful authentication to the authorizer as the user identity for policy assignment lookup. Kerberos-provider allows you to authentication using your kerberos principal and password. Sample: <provider>
<identifier>kerberos-provider</identifier>
<class>org.apache.nifi.kerberos.KerberosProvider</class>
<property name="Default Realm">NIFI.APACHE.ORG</property>
<property name="Authentication Expiration">12 hours</property>
</provider> Bottom line is Apache NiFi does not support managing multiple local accounts. In fact the single-user provider is actually fairly new to Apache NiFi. Prior to its existence, NiFi out-of-the-box started up un-secure over http. Since most browser now force http to https, Apache NiFi added this provider to make it easy for first time user to evaluate the service. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
10-04-2024
12:20 PM
1 Kudo
@Abibee04 Please start a new Community question with the details of your query. I am not clear what you mean by "recover the registry", so the more detail you can provide the better in that new community question by sharing what step you performed and what you are looking to accomplish. Thank you, Matt
... View more
10-04-2024
09:50 AM
1 Kudo
@varungupta Tell me a bit more about your ListFile processor configuration. Is the input directory a shared mount across all your nodes or does each node have a unique set of files in the input directory? Is ListFile configured for execution of "All nodes" or "Primary node"? How many files are being listed when ListFile is executing (Is it less then 10,000)? How often is ListFile scheduled to run? Is List File traversing sub-directories for files? I assume you are extracting the sequenceNo from the filename. As far as Enforce Order goes: How are you handling the various relationships (routing via connection to where)? Are you seeing FlowFiles routed to "overlook" relationship? Matt
... View more
10-03-2024
08:51 AM
@varungupta The EnforceOrder processor maintain state locally specific to each node in the cluster. When you right click on the processor an view state, you are seeing the reported retained local state form each node. The purge of state will only happen on a specific node where the criteria necessary for purge has been met. Might be helpful to share your processor configuration here including properties and scheduling. How are the Source FlowFiles being ingested and fed to this processor? You can look at lineage on the EnforceOrder processor to see FlowFile history. Thanks, Matt
... View more
10-03-2024
08:37 AM
@Axmediko You'll need to provide more detail around your use case. What is contained in your CSV? (sample CSV). What does the URL format look like? (Sample URL showing what portions need to be dynamically set using content from the CSV. The solution here is that you'll need to extract necessary values form the CSV into Attributes on the FlowFile. You can then use those FlowFile attributes to dynamically set the HTTP URL in the invokeHTTP at scheduled execution. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
10-02-2024
09:52 AM
1 Kudo
@sha257 In your ldap-identity-provider located in the login-providers.xml file: <provider>
<identifier>ldap-identity-provider</identifier>
<class>org.apache.nifi.registry.security.ldap.LdapIdentityProvider</class>
<property name="Authentication Strategy">LDAPS</property>
<property name="Manager DN">CN=ABC1234,OU=Service Users,OU=User Accounts,DC=corp1,DC=ad1,DC=xyz,DC=net</property>
<property name="Manager Password">xxxx</property>
<property name="TLS - Keystore">/opt/nifi-registry/nifi-registry-current/conf/keystore.jks</property>
<property name="TLS - Keystore Password">xxx</property>
<property name="TLS - Keystore Type">jks</property>
<property name="TLS - Truststore">/opt/nifi-registry/nifi-registry-current/conf/truststore.jks</property>
<property name="TLS - Truststore Password">xxxx</property>
<property name="TLS - Truststore Type">JKS</property>
<property name="TLS - Client Auth">NONE</property>
<property name="TLS - Protocol">TLS</property>
<property name="TLS - Shutdown Gracefully">false</property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldaps://example-v3-prd.corp1.ad1.xyz.net:636</property>
<property name="User Search Base">OU=User Accounts,DC=corp1,DC=ad1,DC=xyz,DC=net</property>
<property name="User Search Filter">(cn={0})</property>
<property name="Identity Strategy">USE_DN</property>
<property name="Authentication Expiration">12 hours</property>
</provider> I would change "Identity Strategy" from "USE_DN" to "USE_USERNAME". At the NiFi login you are logging in with your ldap CN user string. That user string gets substituted in place of the "{0}" in the "User Search Filter. So you'll want that same string entered at login to be your user identity. This will allow your user to map correctly with the user identity returned by your ldap-user-group-provider. In the file-access-policy-provider you'll want to set the "Initial Admin Identity" to the username you login with rather then the full DN. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
10-02-2024
09:39 AM
1 Kudo
@sha257 There is a lot shared here and I see numerous configuration issues and have some questions. ----> authorizers.xml <---- It is easiest to read the authroizers.xml from bottom up (the bottom of the config file will have the authorizer: <authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers> You can see the authorizer above is loading policies from the "file-access-policy provider", so we scroll up to find that: <accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">CN=ABC1234,OU=Service Users,OU=User Accounts,DC=corp1,DC=ad1,DC=xyz,DC=net</property>
<property name="Nifi Identity 1">CN=ABC1234,OU=Service Users,OU=User Accounts,DC=corp1,DC=ad1,DC=xyz,DC=net</property>
<property name="Nifi Identity">CN=ABC1234,OU=Service Users,OU=User Accounts,DC=corp1,DC=ad1,DC=xyz,DC=net</property>
<property name="NiFi Group Name"/>
<!--<property name="NiFi Identity 1"></property>-->
</accessPolicyProvider> *** This provider will create the authorizations.xml file ONLY if it does not already exist. *** Here we see you have configured your "Initial Admin Identity" with the full DN for your user. This full DN string will be used for setting up the required authorizations for this user Identity to be able to access NiFi and perform administrative tasks (setup new user authorizations, remove access, setup admin level configurations, etc.). Keep in mind that user identities are case sensitive in NiFi. We can also see what provider is being used to provide the current list of user and group identities upon which policies can be set. Here we see it is configured to use "file-user-group-provider". Question: Why do you have the same NiFi host certificate DN configured twice in NiFi Identity and NiFi Identity 1? Now we scroll up to the file-user-group-provider: <userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Initial User Identity 1">CN=ABC1234,OU=Service Users,OU=User Accounts,DC=corp1,DC=ad1,DC=xyz,DC=net</property>
<property name="Initial User Identity admin">cn=CEF567,ou=Solid Users,ou=User Accounts,dc=corp1,dc=ad1,dc=xyz,dc=net</property>
</userGroupProvider> *** This provider will create the users.xml file ONLY if it does not already exist. *** This is one of several providers that can provide initial user identity strings and group identity strings to the configured policy provider for setting up access policies. Here we also see that you used the DN for your "initial user identity admin". While your authorizers.xml has a composite-user-group-provider and ldap-user-group-provider also configured, they are not being used since there is not configuration path to them from the authorizer. Question: Does your LDAP contain entries for your NiFi hosts (not common)? If it does not you will still need the file-user-group-provider to provide those host user identities. Let's assume you need both "file-user-group-provider" and "ldap-user-group-provider". Your authorizers.xml would need to have this structure: Managed-authorizer (configured to use file-access-policy-provider) ---> file-access-policy-provider (configured to use composite-configurable-user-group-provider) ---> composite-configurable-user-group-provider (configured to use both configurable provider: file-user-group-provider and user group provider: ldap-user-group-provider). Then you need both file-user-group-provider and ldap-user-group-provider configured. *** Note: You can NOT have multiple user group providers return the same user-identity or group identity strings. So you'll need to remove the initial admin user from the file-user-group-provider if that admin identity is now going to be returned by the ldap-user-group-provider. Now when it comes to the configuration of your ldap-user-group-provider, that is difficult for me to help with without sample ldap entries for a user and group. But I'll try to comment on what you shared: <userGroupProvider>
<identifier>ldap-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider</class>
<property name="Authentication Strategy">LDAPS</property>
<property name="Manager DN">CN=ABC1234,OU=Service Users,OU=User Accounts,DC=corp1,DC=ad1,DC=xyz,DC=net</property>
<property name="Manager Password">xxx</property>
<property name="TLS - Keystore">/opt/nifi-registry/nifi-registry-current/conf/keystore.jks</property>
<property name="TLS - Keystore Password">xxx</property>
<property name="TLS - Keystore Type">jks</property>
<property name="TLS - Truststore">/opt/nifi-registry/nifi-registry-current/conf/truststore.jks</property>
<property name="TLS - Truststore Password">xxx</property>
<property name="TLS - Truststore Type">JKS</property>
<property name="TLS - Client Auth">NONE</property>
<property name="TLS - Protocol">TLS</property>
<property name="TLS - Shutdown Gracefully">false</property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldaps://example-v3-prd.corp1.ad1.xyz.net:636</property>
<property name="Page Size"/>
<property name="Sync Interval">30 mins</property>
<property name="Group Membership - Enforce Case Sensitivity">false</property>
<property name="User Search Base">OU=User Accounts,DC=corp1,DC=ad1,DC=xyz,DC=net</property>
<property name="User Object Class">person</property>
<property name="User Search Scope">ONE_LEVEL</property>
<property name="User Search Filter">(memberof=CN=AB-C-DEF-ADMIN,OU=Groups,DC=corp1,DC=ad1,DC=xyz,DC=net)</property>
<property name="User Identity Attribute"/>
<property name="User Group Name Attribute"/>
<property name="User Group Name Attribute - Referenced Group Attribute"/>
<property name="Group Search Base"/>
<property name="Group Object Class">group</property>
<property name="Group Search Scope">ONE_LEVEL</property>
<property name="Group Search Filter"/>
<property name="Group Name Attribute"/>
<property name="Group Member Attribute"/>
<property name="Group Member Attribute - Referenced User Attribute"/>
</userGroupProvider> Based on the full DN you configured in the file-user-group-provider, your configuration above is not going to return this user: cn=CEF567,ou=Solid Users,ou=User Accounts,dc=corp1,dc=ad1,dc=xyz,dc=net This is because you have "User Search Base = OU=User Accounts,DC=corp1,DC=ad1,DC=xyz,DC=net" and "User Search Scope = ONE_LEVEL". Your user resides at a SUBTREE level under OU=Solid Users", so the User Search scope needs to be SUBTREE instead. I see that you did not set the User Identity Attribute: this is typically set to the user identity you want to use for the returned user ldap entries. For AD this is typically "sAMAccountName" and for LDAP, it is commonly "cn". Based on your configuration, you are trying to return all AD/LDAP users that are a memberOf one specific group DN. I would recommend also setting the "Group Name Attribute to "cn", so that that string is used as the group identity in NiFi rather then the full group DN. I'll stop here since this is a lot just related to the authorizers.xml file configuration. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-30-2024
12:21 PM
@Leo3103 Here are some additional challenges you may have: NiFi flow definitions do not export with any encrypted values. So it your dataflow uses a component that has sensitive properties (passwords), those will not exist in the flow.json you export. You could get the encrypted values ( they will look like enc{....<long string>....}) from the NiFi flow.json.gz file and add them in to your flow.json.raw produced by the toolkit. In order for your MiNiFi to be able to load a flow.json.raw containing sensitive encrypted values, the same sensitive.props.key value and sensitive.props.algorithm used in your NiFi (nifi.properties) must be used in your MiNiFi bootstrap.conf. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-30-2024
12:09 PM
@Leo3103 Things to note. Your process group that you will be exporting a flow definition for can NOT contain local input or output ports at the top level. ---> The Process Group will become the "root" process group in your MiNiFi deployment. Only "remote" input or output ports are supported at the root level. Within the Process group you are exporting, you can have child process group deeper down that contain local input and output ports. But the test process group "internalTest" you are exporting has a local output port. If you take entire flow.json.gz from the NiFi conf directory and rename it to flow.json.raw, you can start MiNiFi with no issues. Since you are exporting a flow definition of a process group, you'll need to utilize the MiNiFi toolkit to transform it in to the proper format that can be loaded by MiNiFi. The MiNiFi-toolkit can be downloaded from here: https://nifi.apache.org/download/ (select "MINIFI' and click download link for Toolkit). Execute: ./minifi-toolkit/bin/config.sh transform-nifi <exported flow definition> flow.json.raw Now edit the flow.json.raw file and edit the following property at start of file (value can not be 0.) "maxTimerDrivenThreadCount":5 Now you can start your MiNiFi and it will create the flow.json.gz as it starts. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-27-2024
12:17 PM
1 Kudo
@Techie123 Different processor are built to execute differently when scheduled to execute by different Apache NiFi contributors. Some ways to overcome this are to possibly: Adjust the run duration so that processor runs longer when scheduled to process more FlowFiles or produce more FlowFile from an internal queue in a single scheduled execution. Configure your cron to execute multiple times within a short period of time. so maybe every second for 15 seconds within a specific hour and minute. Combination of both the above. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more