Member since
07-30-2019
3472
Posts
1642
Kudos Received
1020
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 249 | 06-03-2026 06:06 PM | |
| 525 | 05-06-2026 09:16 AM | |
| 1025 | 05-04-2026 05:20 AM | |
| 582 | 05-01-2026 10:15 AM | |
| 697 | 03-23-2026 05:44 AM |
04-05-2023
11:30 AM
@davehkd Keep in mind that you are not adding users to either NiFi or NiFi-Registry that can be used for authentication. You are only adding "user identities" that match the user Identity that is returned post successful authentication. For example: Your current method of authentication is via a mutual TLS exchange. Once you pass your client certificate NiFi uses the DN from that certificate as the "user Identity" which is looked up locally to see which authorizations have been associated with that "user identity" (case sensitive). You can additional "user identities" manually right now via the NiFi and NiFi-Registry UI. Then you can assign authorizations to those newly added "user identities". You can use the NiFi TLS toolkit to also create a certificate for each additional user to use for authentication via the mutual TLS exchange. Many users of NiFi and NiFi-Registry use alternate methods of user authentication with "ldap-provider" as probably the most common. If you set up ldap-provider in the login-identity-providers.xml, you would be able to authenticate users via that ldap through a login window presented by NiFi instead of needing to use TLS certificates. You can then optionally, also add the ldap-user-group-provider to the configuration in the authorizers.xml to sync "user identities" and "group identities" from your ldap server so you do not need to manually add these identities yourself via the UI. Hope this clarifies some things for you. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
04-04-2023
01:20 PM
@Meeran Where are you seeing above exception? System Error
The request contained an invalid host header [internal-myappname-13718787520.us-east-1.elb.amazonaws.com:8443] in the request [/]. Check for request manipulation or third-party intercept. In your Load Balancer logs? This host header "internal-myappname-13718787520.us-east-1.elb.amazonaws.com" is in your nifi.web.proxy.host list or in the SAN list within each of your node's certificates? In NiFi, check the nifi-user.log and nifi-app.log for any WARN or ERROR log output being produced when you try to access NiFi via the load balancer URL.
... View more
04-04-2023
12:41 PM
1 Kudo
@Dave0x1 Apache NiFi takes carful consideration when upgrading/migrating from an older version to a newer version with in reason. Like upgrading from 1.13 to 1.19 would be expected to pose no real issues; however, jumping from 1.10 directly to 1.19 may pose some challenges. That being said, rolling back is not something that can be guaranteed to go smoothly as some versions can result in changes to database structure/content, modification/change to NiFi components (processor, controller services, reporting tasks, etc). In a best case scenario, a rollback my result in a working NiFi, with unexpected unexpected invalid components on the NiFi canvas. This happens when new properties are introduced in components being used, Those component changes get written to the flow.xml.gz and new flow.josn.gz files. So on rollback those new properties turn in to dynamic properties which may result in an invalid component until they are removed manually. In other scenarios, there may have been upgrades/changes to H2 DB that won't rollback smoothly. So perhaps taking a backup of the flow.xml.gz. flow.json.gz, and database_repository/ directory contents would be useful in the vent of rollback. Good news is that there is nothing in the H2 DBs located in the database_repository you need to retain. The nifi-flow-audit DB tracks history of changes you made on the canvas. The other user and identity dbs can just be removed and allowed to be recreated on next startup. So in your rollback test, I would rename the existing DB files and restart allowing new to be created. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
04-04-2023
12:13 PM
@davehkd Sounds like your NiFi-Registry service install went well. There is not such thing as a NiFi registry tool. I assume you were referring to the NiFi TLS toolkit you used to generate your certificates and keystores? You are dealing with an issue related to your initial admin. 1. When you configured the user-group-provider and file-access-policy provider, you stated that you used your "ec2-user" DN; however, the authorization exception you shared is as follows" AccessDeniedExceptionMapper: identity CN=ec2-user does not have permission to access the requested resource. That is not a the user's full DN in that exception. To NiFi and NiFi-Registry the user identity string "CN=ec2-user. OU=NiFi" is not the same user as "CN=ec2-user". What appears to me from your description is the full user DN has been created in the users.xml and authorization for that full DN setup in the authorizations.xml. So to answer why your "CN=ec2-user, OU=NiFi" DN resulted in NiFi-Registry seeing your user identity as only "CN=ec2-user" comes down to identity mapping properties. So in yoou nifi-registry.properties file you'll want to look for sets of properties like the following: nifi.registry.security.identity.mapping.pattern.<some string>=<some regex pattern with 1 or more capture groups>
nifi.registry.security.identity.mapping.value.<some string>=$1
nifi.registry.security.identity.mapping.transform.<some string>=NONE My guess here is you will find some regex pattern that matches on your user's full DN and the corresponding value property is then only returning "CN=ec2-user". You can remove or modify the matching mapping pattern or change your initial admin to just "CN=ec2-user" instead of using full DN. Either will work. Keep in mind if you decide to go with "CN=ec2-user", then you will need edit the authorizers.xml to use "CN=ec2-user" instead of full DN. You will also need to remove the current users.xml and authorizations.xml files created by the authorizer on first startup so they get recreated with these new identity strings. The authorizer providers only generate the authorizations.xml and users.xml on startup if they do NOT already exist. 2. the following output you reported from your nifi-registry-app.log is normal and expected: Kerberos service ticket not supported by the NiFi Registry. This is not an ERROR log. It is simply reporting that the spnego related kerberos properties in the nifi-registry.properties file were not configured and thus kerberos authentication via spnego is not supported. You can just ignore this notification. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
03-30-2023
02:44 PM
1 Kudo
@wffger2 @cotopaul The InstanceIdentifier was introduced when NiFi switched from using the flow.xml.gz to the flow.josn.gz for flow persistence. It is also currently being written to the flows stored in the NiFi-Registry. I say "currently" because this jira exist and is still open: https://issues.apache.org/jira/browse/NIFI-11321 So as @cotopaul points out, the "instanceIdentifier" will be unique for each instantiation of the flow definition, but the "identifier" will be the same. Downside here is every time you want to do a comparison, you'll need to download latest flow definition from DEV and UAT environments and compare for difference based on the Identifiers. Thanks, Matt
... View more
03-30-2023
01:37 PM
@nabilraza456 I am curious about the solution you said worked in HDF? Can you share that with me? I'd like to explore how that capability changed going to CFM. Thanks, Matt
... View more
03-30-2023
01:31 PM
@cotopaul Is that complete stack trace from the nifi-app.log? What version of Apache NiFi? What version of Java? Have you tried using ConsumeKafkaRecord processor instead of ConsumeKafka --> MergeContent? Do you have issue only when using the ParquetRecordSetWriter? How large are the FlowFiles coming out of the MergeContent processor? Have you tried reducing the size of the Content being output from MergeContent processor? Thanks, Matt
... View more
03-30-2023
01:13 PM
@saquibsk Stop your NiFi service. Try using the following command to reset the single user "username" and "password" to new values: $ ./bin/nifi.sh set-single-user-credentials <username> <password> Then start your NiFi service and see it you can successfully login. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
03-30-2023
08:58 AM
2 Kudos
@Breezer NiFi was historically never built to manage local users. NiFi provides no mechanism for creating and managing multiple users locally. That being said, the Apache NiFi community found that many new users to NiFi were simply starting up unsecure NiFi instances on publicly accessible networks and decided to make changes so that by default NiFi would start with out of the box configuration secured over https. This change was released as part of the Apache NiFi 1.14 release and involved the following changes to make this work. 1. NiFi toolkit is used automatically to generate a keystore and truststore using self signed certs to secure NiFi. 2. A secured NiFi will require users/clients to authenticate and be authorized to interact with the NiFi UI in various ways. This means that out of the box there would need to be an authorizer and a means to define some user that could then be auto authorized to the needed policies. These changes were all part of https://issues.apache.org/jira/browse/NIFI-8220 The single-user-authorizer and single-user-provider were never intended for use in production as they do not provide granular multi-user level of authentication and authorization (which is what you are looking for). The simply provide for a single user who is authorized to every NiFi policy allowing for a secured environment out of the box. Since NiFi never has and does not have any intention of managing users locally (creating multiple local users with passwords managed through NiFi UI) in the future, you'll need to utilize one of the other available user authentication methods if you want an environment which supports multiple users with unique authorizations. Those methods are explained here: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#user_authentication I see you don't want to rely on some external authentication provider like ldap, kerbersos, knox, etc. and that is fine. User authentication can also be achieved via a mutual TLS handshake. All this requires is generating a unique user certificate for each of your 3 users. A basic setup like this would require you to configure your NiFi to use the follwoing: Authentication: Clear the "single-user-provider" for the "nifi.security.user.login.provider" property in the nifi.properties file. Use the NiFi TLS toolkit to generate your certifcates: https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#tls_toolkit. Or you could use an external free certificate provider like Tinycert to create a certificate for each your NiFi instance(s) and a certifcate for each of your users. Authorization: Change the "single-user-authorizer" for the " nifi.security.user.authorizer" property in the nifi.properties file to "managed-authorizer". Build a new authorizers.xml that uses the "managed-authorizer" (https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#standardmanagedauthorizer), "file-access-policy-provider" (https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#fileaccesspolicyprovider), and "file-user-group-provider" (https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#fileusergroupprovider). Example: Authorizers.xml configuration: <authorizers>
<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<property name="Initial User Identity 1"><full DN from user certifcate 1></property>
<property name="Initial User Identity 2"><full DN from user certifcate 2></property>
<property name="Initial User Identity 3"><full DN from user certifcate 3></property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity"><full DN from user certificate 1></property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1"></property>
</accessPolicyProvider>
<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers> This authorizers.xml setup will add your three user identities to NiFI for purpose of authorizing them against NiFi policies only. One of those users will be designated as the "initial admin" in the file-access-policy-provider. This user will be assigned to the required policies needed for that user to act as admin. That admin user can then access NiFi and setup authorization policies for the other two users. The Certificates created for your users would be provided to each user. The user can then load that certificate into their browser. When the user navigates the the HTTPS NiFi URL, NiFi will request that client provide a certifcate and the loaded certificate can be used. This handles the unique user authentication. More details on setting up additional authorization policies for yoru users can be found here: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#config-users-access-policies If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
03-30-2023
05:52 AM
@davehkd Changes committed to an early version 1.19 will persist into the next version unless a specific closed jiras exists that makes another changes impacting version. I saw no other newer jiras related to Kotlin version changes at time of this response. Matt
... View more