Member since
07-30-2019
3470
Posts
1642
Kudos Received
1018
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 275 | 05-06-2026 09:16 AM | |
| 457 | 05-04-2026 05:20 AM | |
| 335 | 05-01-2026 10:15 AM | |
| 517 | 03-23-2026 05:44 AM | |
| 390 | 02-18-2026 09:59 AM |
04-04-2023
12:13 PM
@davehkd Sounds like your NiFi-Registry service install went well. There is not such thing as a NiFi registry tool. I assume you were referring to the NiFi TLS toolkit you used to generate your certificates and keystores? You are dealing with an issue related to your initial admin. 1. When you configured the user-group-provider and file-access-policy provider, you stated that you used your "ec2-user" DN; however, the authorization exception you shared is as follows" AccessDeniedExceptionMapper: identity CN=ec2-user does not have permission to access the requested resource. That is not a the user's full DN in that exception. To NiFi and NiFi-Registry the user identity string "CN=ec2-user. OU=NiFi" is not the same user as "CN=ec2-user". What appears to me from your description is the full user DN has been created in the users.xml and authorization for that full DN setup in the authorizations.xml. So to answer why your "CN=ec2-user, OU=NiFi" DN resulted in NiFi-Registry seeing your user identity as only "CN=ec2-user" comes down to identity mapping properties. So in yoou nifi-registry.properties file you'll want to look for sets of properties like the following: nifi.registry.security.identity.mapping.pattern.<some string>=<some regex pattern with 1 or more capture groups>
nifi.registry.security.identity.mapping.value.<some string>=$1
nifi.registry.security.identity.mapping.transform.<some string>=NONE My guess here is you will find some regex pattern that matches on your user's full DN and the corresponding value property is then only returning "CN=ec2-user". You can remove or modify the matching mapping pattern or change your initial admin to just "CN=ec2-user" instead of using full DN. Either will work. Keep in mind if you decide to go with "CN=ec2-user", then you will need edit the authorizers.xml to use "CN=ec2-user" instead of full DN. You will also need to remove the current users.xml and authorizations.xml files created by the authorizer on first startup so they get recreated with these new identity strings. The authorizer providers only generate the authorizations.xml and users.xml on startup if they do NOT already exist. 2. the following output you reported from your nifi-registry-app.log is normal and expected: Kerberos service ticket not supported by the NiFi Registry. This is not an ERROR log. It is simply reporting that the spnego related kerberos properties in the nifi-registry.properties file were not configured and thus kerberos authentication via spnego is not supported. You can just ignore this notification. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
03-30-2023
02:44 PM
1 Kudo
@wffger2 @cotopaul The InstanceIdentifier was introduced when NiFi switched from using the flow.xml.gz to the flow.josn.gz for flow persistence. It is also currently being written to the flows stored in the NiFi-Registry. I say "currently" because this jira exist and is still open: https://issues.apache.org/jira/browse/NIFI-11321 So as @cotopaul points out, the "instanceIdentifier" will be unique for each instantiation of the flow definition, but the "identifier" will be the same. Downside here is every time you want to do a comparison, you'll need to download latest flow definition from DEV and UAT environments and compare for difference based on the Identifiers. Thanks, Matt
... View more
03-30-2023
01:37 PM
@nabilraza456 I am curious about the solution you said worked in HDF? Can you share that with me? I'd like to explore how that capability changed going to CFM. Thanks, Matt
... View more
03-30-2023
01:31 PM
@cotopaul Is that complete stack trace from the nifi-app.log? What version of Apache NiFi? What version of Java? Have you tried using ConsumeKafkaRecord processor instead of ConsumeKafka --> MergeContent? Do you have issue only when using the ParquetRecordSetWriter? How large are the FlowFiles coming out of the MergeContent processor? Have you tried reducing the size of the Content being output from MergeContent processor? Thanks, Matt
... View more
03-30-2023
01:13 PM
@saquibsk Stop your NiFi service. Try using the following command to reset the single user "username" and "password" to new values: $ ./bin/nifi.sh set-single-user-credentials <username> <password> Then start your NiFi service and see it you can successfully login. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
03-30-2023
08:58 AM
2 Kudos
@Breezer NiFi was historically never built to manage local users. NiFi provides no mechanism for creating and managing multiple users locally. That being said, the Apache NiFi community found that many new users to NiFi were simply starting up unsecure NiFi instances on publicly accessible networks and decided to make changes so that by default NiFi would start with out of the box configuration secured over https. This change was released as part of the Apache NiFi 1.14 release and involved the following changes to make this work. 1. NiFi toolkit is used automatically to generate a keystore and truststore using self signed certs to secure NiFi. 2. A secured NiFi will require users/clients to authenticate and be authorized to interact with the NiFi UI in various ways. This means that out of the box there would need to be an authorizer and a means to define some user that could then be auto authorized to the needed policies. These changes were all part of https://issues.apache.org/jira/browse/NIFI-8220 The single-user-authorizer and single-user-provider were never intended for use in production as they do not provide granular multi-user level of authentication and authorization (which is what you are looking for). The simply provide for a single user who is authorized to every NiFi policy allowing for a secured environment out of the box. Since NiFi never has and does not have any intention of managing users locally (creating multiple local users with passwords managed through NiFi UI) in the future, you'll need to utilize one of the other available user authentication methods if you want an environment which supports multiple users with unique authorizations. Those methods are explained here: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#user_authentication I see you don't want to rely on some external authentication provider like ldap, kerbersos, knox, etc. and that is fine. User authentication can also be achieved via a mutual TLS handshake. All this requires is generating a unique user certificate for each of your 3 users. A basic setup like this would require you to configure your NiFi to use the follwoing: Authentication: Clear the "single-user-provider" for the "nifi.security.user.login.provider" property in the nifi.properties file. Use the NiFi TLS toolkit to generate your certifcates: https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#tls_toolkit. Or you could use an external free certificate provider like Tinycert to create a certificate for each your NiFi instance(s) and a certifcate for each of your users. Authorization: Change the "single-user-authorizer" for the " nifi.security.user.authorizer" property in the nifi.properties file to "managed-authorizer". Build a new authorizers.xml that uses the "managed-authorizer" (https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#standardmanagedauthorizer), "file-access-policy-provider" (https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#fileaccesspolicyprovider), and "file-user-group-provider" (https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#fileusergroupprovider). Example: Authorizers.xml configuration: <authorizers>
<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<property name="Initial User Identity 1"><full DN from user certifcate 1></property>
<property name="Initial User Identity 2"><full DN from user certifcate 2></property>
<property name="Initial User Identity 3"><full DN from user certifcate 3></property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity"><full DN from user certificate 1></property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1"></property>
</accessPolicyProvider>
<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers> This authorizers.xml setup will add your three user identities to NiFI for purpose of authorizing them against NiFi policies only. One of those users will be designated as the "initial admin" in the file-access-policy-provider. This user will be assigned to the required policies needed for that user to act as admin. That admin user can then access NiFi and setup authorization policies for the other two users. The Certificates created for your users would be provided to each user. The user can then load that certificate into their browser. When the user navigates the the HTTPS NiFi URL, NiFi will request that client provide a certifcate and the loaded certificate can be used. This handles the unique user authentication. More details on setting up additional authorization policies for yoru users can be found here: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#config-users-access-policies If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
03-30-2023
05:52 AM
@davehkd Changes committed to an early version 1.19 will persist into the next version unless a specific closed jiras exists that makes another changes impacting version. I saw no other newer jiras related to Kotlin version changes at time of this response. Matt
... View more
03-30-2023
05:49 AM
@wffger2 NiFi Flow Definitions are meant to be reusable snippets of flow. Meaning it can be imported to the canvas of the same NiFi Cluster or to the canvas of another NiFi cluster. Since you can not have multiple components on the canvas using the same component UUID, you will not be able to compare Flow Definitions between two different environments without seeing differences in those UUIDs. If you want to track differences in dataflows between your UAT and DEV environments, NiFi-Registry is the method you want to use. You can install a single NiFi-Registry and configure both your NiFi's to point at that same instance of NiFi-Registry. Then on your DEV environment you start version control on a process group (testNiFiDefinition). Then on your UAT environment you import that same version flow from the NiFi-registry. Now you have both your NiFi environments tracking against the same version flow stored in that NiFi-registry. While the component UUID will still be different, they both track to same versioned flow. If you then make changes in DEV and commit those to NiFi-Registry as a new version of the already version controlled flow, you UAT environment will report a newer version as being available. You can see the differences directly from the NiFi UI and easily upgrade your dataflow on the UAT environment to the newer version available if you choose. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
03-30-2023
05:35 AM
3 Kudos
@Meeran The solution here is to setup an external load balancer in front of your 3 NiFi nodes. Then have your clients point at that load balancer. The Load Balancer would then be responsible for sending your client request to one of the available NiFi nodes. (If a node goes down, the Load Balancer simply does not send client requests to that node. When using a Load Balancer in front of your NiFi cluster, it is important that the Load Balancer is configured to use sticky sessions. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
03-27-2023
11:12 AM
@srilakshmi The PublishKafka processor can be configured with a comma separated list of KaFla brokers. If the processor at time of execution is able to communicate with one of these configured brokers, it will received a destination for publishing the content. If during the publish a failure occurs, The FlowFile is routed to the failure relationship. You have configurable options to retry on failure x number of times. You should avoid auto-terminating failure relationships in your datafow designs unless data loss is acceptable. Each attempt is a new execution of the processor which means connect to broker again. A failure is when PublishKafka was unable to send all the content bytes (for example: connection gets closed). Best Effort and Guarantee single node delivery setting in the PublishKafka processor have nothing to do with the NiFi nodes in the NiFi cluster. This has to do with the nodes in the destination Kafka setup. In a NiFi cluster each node executes its own copy of the dataflow(s) and each node has its own content and FlowFile repositories. Nodes are unaware of FlowFiles that exist on the other nodes in the cluster. So a FlowFile's content that fails to publish on say node2 will route to failure relationship on node 2 and if you use retry, will be executed on again on node 2. When a node goes down the FlowFile queued in connection remain on that node until it is brought back on line. When the node comes back up, it will continue processing FlowFiles from the last connection in which they were queued. So it is important the the Content and FlowFile repositories are protected to avoid dataloss (such as using RAID storage). A node that is disconnected from the cluster will still execute its dataflow(s) as long as NiFi is still running on that node. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more