Member since
07-30-2019
3369
Posts
1615
Kudos Received
996
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 109 | 10-08-2025 10:52 AM | |
| 87 | 10-08-2025 10:36 AM | |
| 166 | 10-03-2025 06:04 AM | |
| 123 | 10-02-2025 07:44 AM | |
| 290 | 09-23-2025 10:09 AM |
06-05-2025
09:30 AM
Hello @agriff Welcome to the community. I find it very odd that no ERROR logging is being produced when your PublishKafka processor is routing FlowFiles to the Failure relationship. Keep in mind that NiFi does not control the logging in the third party libraries that a given processor type may use. So it is possible that the Kafka client library for the specific PublishKafka version has not DEBUG logging. This is not specific to only publishKafka, but can be the case for any processor component that is dependent on a third party client library to which the open source community has no ability to modify. The "PublishKafka" with no version number in its name is the oldest of all the client versions. It was the first built and was deprecated some time ago because of its version age. If you Kafka server is newer then 0.8, you'll want to be using a different version of this processor. There are so many version of the Kafka based processors because of version client/server incompatibility between versions. What version of Kafka are you publishing to? What version of Apache NiFi are you using? Which PublishKafka processor are you using? There are numerous that use different Kafka Client library versions. You'll want to use the appropriate one that aligns with your Kafka server version? Changing the "bulletin level" within the processor has not affect on the log level for that processor class in the nifi-app.log. To set this processor class to DEBUG in the nifi-app.log, you'll need to modify the logback.xml in the NiFi conf directory: Example logger line you would add to logback.xml ith rest of existing loggers: <logger name="org.apache.nifi.processors.kafka.pubsub.PublishKafka" level="DEBUG"/> The class name will vary by processor: org.apache.nifi.processors.kafka.pubsub.PublishKafka org.apache.nifi.processors.kafka.pubsub.PublishKafka_0_10 org.apache.nifi.processors.kafka.pubsub.PublishKafka_0_11 org.apache.nifi.processors.kafka.pubsub.PublishKafka_1_0 org.apache.nifi.processors.kafka.pubsub.PublishKafka_2_0 org.apache.nifi.processors.kafka.pubsub.PublishKafka_2_6 org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_0_10 org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_0_11 org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_1_0 org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_2_0 org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_2_6 Sharing your dataflow and processor configuration might also be helpful to your query. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
06-04-2025
06:43 AM
@Artem_Kuzin I suggest starting with logging in to Ranger UI and verifying under "Audit" --> "Plugin Status" that your HDFS and Hive services are reported as having downloaded and made active the latest updated policies. If they have not, I start checking the HDFS and Hive logs for any logging related to issues connecting or fetching policies json from Ranger. Beyond above, I'd recommend that you open a support case with Cloudera (assuming you have a valid support license) where you can securely share your configuration and logs for more in-depth troubleshooting assistance with this issue. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
06-03-2025
01:08 PM
@lynott It sounds like you created wildcard certificates toi use with your NiFi and NiFi-Registry services/instances. I strongly discourage this from a security standpoint. NiFi utilizes its certificates to perform both clientAuth and ServerAuth. When used to perform clientAuth, such as connect to NiFi-Registry, the clientAuth CN is presented as the client/user identifier. This means the server side (NiFi-Registry in this use case) would need to authorize that wildcard CN which exposes your NiFi-Registry to unauthorized access by any clientAuth certificate that matches against that wildcard authorized entity (*.example.com). While I recommend and encourage creation off unique certificates per node with a shared common name as one of the SAN entries as a security best practice. It is possible to create once certificate that contains SAN entries for every host on which hat certificate is used. However, the CN in that one certificate should not use wildcards, so that authorizations can not resolve to any other value then the expected non wildcard CN. Looking back at your nifi-registry.properties file, I can se that you have not https://nifi.apache.org/nifi-docs/administration-guide.html#identity-mapping-properties configured. nifi.registry.security.identity.mapping.pattern.<somestring>=
nifi.registry.security.identity.mapping.value.<somestring>=
nifi.registry.security.identity.mapping.transfrom.<somestring>= Identity Mapping Properties are used to manipulated user/client identities post authentication and pre-authorization. Without using identity.mapping.patterns, the complete clientAuth DistinquishedName (DN) will be passed to the authorizer in NiFi where it is expected to be authorized to Can Proxy Request (read, write, delete) and Can Manage Buckets (Read). Since you aer are using a wildcard DN, authorizing the NiFi node hostnames in NiFi-Registry is going to accomplish nothing since those hostnames can't be extarcted from the DN for authorization. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
06-03-2025
05:55 AM
@lynott @sydney- Are you good now? Your last response contains no questions. If my previous response(s) helped with your initial issue, please a take a moment to click accepted on those responses. Thank you, Matt
... View more
06-03-2025
05:48 AM
1 Kudo
@shiva239 NiFi's DBCPConnectionPool controller services is designed to create a pool of connections that are created the first time it is invoked. These connections can then be used by multiple components on the canvas that are configured to use this same connection pool. This is designed to maximize performance of the dataflows. You can control the behavior of the connection pool using the configuration properties available to this controller service: Max Idle Connections Minimum Evictable Idle Time Minimum Idle Connections Soft Minimum Evictable Idle Time Time Between Eviction Runs Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
06-02-2025
05:18 AM
1 Kudo
@asand3r Your issue is caused by a misconfiguration in the authorizers.xml file here: <userGroupProvider>
<identifier>composite-configurable-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.CompositeUserGroupProvider</class>
<property name="User Group Provider 0">file-user-group-provider</property>
<property name="User Group Provider 1">ldap-user-group-provider-1</property>
<property name="User Group Provider 2">ldap-user-group-provider-2</property>
<property name="User Group Provider 3">ldap-user-group-provider-3</property>
<property name="User Group Provider 4">ldap-user-group-provider-4</property>
</userGroupProvider> The wrong "class" is being used and the wrong property name is being used for the file-user-group-provider. It should look like this: <userGroupProvider>
<identifier>composite-configurable-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.CompositeConfigurableUserGroupProvider</class>
<property name="Configurable User Group Provider">file-user-group-provider</property>
<property name="User Group Provider 1">ldap-user-group-provider-1</property>
<property name="User Group Provider 2">ldap-user-group-provider-2</property>
<property name="User Group Provider 3">ldap-user-group-provider-3</property>
<property name="User Group Provider 4">ldap-user-group-provider-4</property>
</userGroupProvider> The "class" needs to be: org.apache.nifi.registry.security.authorization.CompositeConfigurableUserGroupProvider The above class support one defined "configurable user group provider". A configurable user group provider (file-user-group-provider) is one that allows manual manipulation via the NiFi/NiFi-Registry UI. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
05-30-2025
06:05 AM
@Ripul Welcome to the Cloudera Community! Sharing a screenshot would be helpful here, but I am assume what you are seeing is something like this when you login with your admin user or other users: This is because of an authorization issue. When NiFi is started for the first time it does not have a flow.josn.gz file yet which contains everything you see on the NiFi canvas. So NiFi will generate that flow.josn.gz which will consist of just a root process group. You'll notice on the cavas the above "Operation" panel. It will show the current selected component on the canvas. With nothing selected on the canvas, it will show details for whichever NiFi Process Group you are currently displaying. Since this is a new install, what the Operation panel is showing is this generated root process group. Anytime you see the name as just the UUID for a component, it indicates the currently authenticated user is not authorized to view that component. A greyed out "gear" (configuration) icon indicates user is not authorized to modify the component. A greyed out "key" (Access Policies) icon indicates currently authenticated user is not authorized to view and maybe modify policies (authorizations) in that component. NiFi provides very granular authorization control all the way down to the individual component level. This may sound like a lot to need to manage; however, there is policy inheritance in place. Example: You add a processor to the canvas. If not explicit policy is defined on the processor itself it will inherit policy from the process group it is inside. If there is no policy defined on the process group, it will inherit policy from parent process group. At the very top level is the above mentioned parent process group. So setting policies on the parent process group will control access on everything added to cavas until ab explicit access policy is set on a sub component. There are also global policies that can be setup and your "admin" user should have been setup on a number of these. From the above global menu found in upper right corner you should see that "Policies" is not greyed out for your admin user. Within global "Policies", all users need to be granted "view the user interface" in order to access the user interface, so it sounds like you have already done this for other users. Your "admin" user should also have "access all policies" (view and modify) which allows that user to view and modify access policies (authorizations) on every component anywhere on the canvas. This policy is what makes the "key" icon not greyed out on the "Operation" panel mentioned earlier. So to give select users (including your admin user) the ability to add components to the root process group, your admin user will need to select the key icon on the root process group and grant those users: Once your admin user and other users are properly authorized to "view the component", the Operate panel will show the process group name instead of just the process group assigned UUID. The gear icon will not be greyed out once your admin user and other users have "modify the component". "Modify the component" on a process group will also allow added users to see the component adding icon a the top of the UI. I am not going to cover all the NiFi Policies, but they can be found in the NiFi Administration guide under Configuring Users & Access Policies Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
05-29-2025
12:36 PM
@asand3r It is very common to use a load balancer in front of NiFi's UI, but it is necessary to configured Session Affinity (sticky sessions) in the load balancer to make sure all subsequent requests are directed to same node in same session. When you login to a NiFi node using ldap auth, that specific node issues you a user token and stores a corresponding server side token. That server side token is only present on the one node that handled the user authentication. So without session affinity configured redirects within the same session are likely to end up going to other nodes which will reject the client token. The server token is how NiFi controls access expiration since it is not possible to guarantee a client expires and stops using the client token issued. The NiFi node will remove the server side token at expiration or when user clicks logout. You'll also want to make sure that your NiFi node certificates also include a SAN entry for: nifi-service.company.com Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
05-29-2025
12:21 PM
@lynott I see some issues in your configurations. And have some questions in order to address some other potential issues. Authorizers.xml: File-user-group-provider <identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Initial User Identity 1">employee</property>
</userGroupProvider> Here you are defining manually a "user identity" of "employee". Is this "employee" a user that is authenticating into your NiFi via ldap authentication. If so, I'd expect this user identity to be returned by the ldap-user-group-provider instead of the file-user-group-provider. You can NOT have two user-group-provider return the same user identity. So if "employee" is going to be returned by ldap-user-group-provider, it needs to be removed from file-user-group-provider. NOTE: The file-user-group-provider will only generate the users.xml if it does not already exist. So modifications to the configuration in the file-user-group-provider will NOT result in any changes to an already existing users.xml. the users.xml file will only contain manually added user and group identities. user and group identities returned by other user-group-provider will not get added to the users.xml. ldap-user-group-provider I assume you ldap connection is good, so I'll just focus on the user sync and group sync areas: <property name="User Search Base"></property>
<property name="User Object Class"></property>
<property name="User Search Scope">SUBTREE</property>
<property name="User Search Filter"></property>
<property name="User Identity Attribute">cn</property>
<property name="User Group Name Attribute"></property>
<property name="User Group Name Attribute - Referenced Group Attribute"></property>
<property name="Group Search Base">OU=COMPANYGroups,DC=corp,DC=company,DC=com</property>
<property name="Group Object Class">group</property>
<property name="Group Search Scope">ONE_LEVEL</property>
<property name="Group Search Filter">((cn=nifi-admins))</property>
<property name="Group Name Attribute">cn</property>
<property name="Group Member Attribute">member</property>
<property name="Group Member Attribute - Referenced User Attribute"></property> As configured you are performing a group sync that will return all the "member" attribute values from the "nifi-admins" group. 1. Are the member attribute values full user DNs? Because you have "cn" configured in the user sync section "user Identity Attribute". each returned "member" is going to be looked up in order to get the cn value. Since NiFi and NiFi-registry user and group identities are case sensitive, it is important that the user identities authorized match exactly with the user identity returned by user authentication in to NiFi-Registry. I see from the nifi-registry.properties file that you are using the ldap-provider for user authentication. Assuming you a re logging in using the user's "cn" as the username, you need to make sure that "USE_USERNAME" is being used and not "USE_DN" in that ldap-provider. Composite-provider: <userGroupProvider>
<identifier>composite-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.CompositeUserGroupProvider</class>
<property name="Configurable User Group Provider">file-user-group-provider</property>
<property name="User Group Provider 1">ldap-user-group-provider</property>
</userGroupProvider> Here you are using the "composite-user-group-provider", but are trying to use a configurable provider with it. Instead you should be using: <userGroupProvider>
<identifier>composite-configurable-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.CompositeConfigurableUserGroupProvider</class>
<property name="Configurable User Group Provider">file-user-group-provider</property>
<property name="User Group Provider 1">ldap-user-group-provider</property>
</userGroupProvider> You'll notice the "class" is different between these two providers. The "identifier" is not important, but must match the identifier set in your file-access-policy-provider. file-access-policy-prpovider <identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
<property name="User Group Provider">composite-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">employee</property>
<property name="NiFi Group Name"></property>
<!--<property name="NiFi Identity 1"></property>-->
</accessPolicyProvider The file-access-policy provider uses the authorizations.xml file to store all configured authorizations. It will only generate a authorizations.xml file if one does not already exist. It can be configured with an "initial Admin Identity". This configured Initial Admin Identity must be returned by one of yoru user-group-providers. In your case "employee" is being used here. The "NiFi Identity 1", "NiFi Identity 2", "NiFi Identity <xyz>" can be added to define your NiFi node certificate Identities which would allow this provider to seed the proxy policies needed by the NiFi hosts. But keep in mind that the file-user-group-provider would also need to generate these identities for this to work. ------------------- users.xml and authorizations.xml files: Your users.xml only contains one entry for "employee" which corresponds with your authorizers configuration. That user is assigned a UUID. I can then see that UUID properly authorized as and admin user in the authorizations.xml file. What I do not see in the users.xml file is any group defined with uuid: 8c295cae-a773-4d6a-98cd-eef47d0b8189 2. I assume this is the uuid for your ldap-user-group-provider returned "nifi-admins" group? 3. I assume you manually added this group to the authorizations.xml file since you mentioned you have been unable to manually configure user and policies from within the NiFi-Registry UI yet? ---------------------- NiFi-Registry UI: 4. When you login to the NiFi-Registry using user "employee" do you not see "employee" displayed in upper right corner of UI along with the wrench icon? Remember that only case sensitive "employee" and what ever group happens to be "8c295cae-a773-4d6a-98cd-eef47d0b8189" are currently authorized as an admins. 5. I assume you obtained "8c295cae-a773-4d6a-98cd-eef47d0b8189" by enabling debug in the logback.xml for the ldap-user-group-provider class? Or is this the uuid for the NiFi node certificate derived user identity? Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
05-28-2025
01:37 PM
@lynott When trying to start version control, the NiFi instance will authenticate with NiFi-registry via a MutualTLS exchange. The DN from NiFi's keytore clientAuth privateKey entry is used as the user identity (post NiFi-Registry applying any matching identity.mapping patterns to it). In the request for list of buckets will be the "user identity" displayed in teh upper right corner of NiFi when the request was made. The NiFi node is proxying the request on behalf of that user identity. So If you are not seeing any buckets, either there is a proxy issue for one of your NiFi nodes or the NiFi user identity is not properly authorized in NiFi-Registry for any buckets. Without seeing your NiFi-registry configurations, it would be difficult to say exactly what your issue is. This includes looking at: NiFi-Registry.properties authorizers.xml users.xml authorizations.xml Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more