Member since
07-30-2019
3406
Posts
1623
Kudos Received
1008
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 336 | 12-17-2025 05:55 AM | |
| 397 | 12-15-2025 01:29 PM | |
| 405 | 12-15-2025 06:50 AM | |
| 367 | 12-05-2025 08:25 AM | |
| 604 | 12-03-2025 10:21 AM |
04-06-2023
06:26 AM
@drewski7 Why are you connecting your NiFi to NiFi-Registry via Knox instead of directly[y connecting to the NiFi-registry host? NiFi will authenticate via a mutual TLS exchange using the keystore and truststores configured in nifi.properties and the nifi-registry.properties files. Thanks, Matt
... View more
04-05-2023
11:46 AM
1 Kudo
@RRosa This is likely result of changes on Twitter's side related to changes to their twitter API endpoint and not directly related to the ConsumeTwitter processor included with NiFi: https://techcrunch.com/2023/02/01/twitter-to-end-free-access-to-its-api/ Looks like free access to consume from Twitter was cut off in February. From the following page, we can see that the Free tier now only allows writes and no consumes. https://developer.twitter.com/en/products/twitter-api You would need at least a "Basic" subscription to continue to consume from Twitter. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
04-05-2023
11:30 AM
@davehkd Keep in mind that you are not adding users to either NiFi or NiFi-Registry that can be used for authentication. You are only adding "user identities" that match the user Identity that is returned post successful authentication. For example: Your current method of authentication is via a mutual TLS exchange. Once you pass your client certificate NiFi uses the DN from that certificate as the "user Identity" which is looked up locally to see which authorizations have been associated with that "user identity" (case sensitive). You can additional "user identities" manually right now via the NiFi and NiFi-Registry UI. Then you can assign authorizations to those newly added "user identities". You can use the NiFi TLS toolkit to also create a certificate for each additional user to use for authentication via the mutual TLS exchange. Many users of NiFi and NiFi-Registry use alternate methods of user authentication with "ldap-provider" as probably the most common. If you set up ldap-provider in the login-identity-providers.xml, you would be able to authenticate users via that ldap through a login window presented by NiFi instead of needing to use TLS certificates. You can then optionally, also add the ldap-user-group-provider to the configuration in the authorizers.xml to sync "user identities" and "group identities" from your ldap server so you do not need to manually add these identities yourself via the UI. Hope this clarifies some things for you. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
04-04-2023
01:20 PM
@Meeran Where are you seeing above exception? System Error
The request contained an invalid host header [internal-myappname-13718787520.us-east-1.elb.amazonaws.com:8443] in the request [/]. Check for request manipulation or third-party intercept. In your Load Balancer logs? This host header "internal-myappname-13718787520.us-east-1.elb.amazonaws.com" is in your nifi.web.proxy.host list or in the SAN list within each of your node's certificates? In NiFi, check the nifi-user.log and nifi-app.log for any WARN or ERROR log output being produced when you try to access NiFi via the load balancer URL.
... View more
04-04-2023
12:41 PM
1 Kudo
@Dave0x1 Apache NiFi takes carful consideration when upgrading/migrating from an older version to a newer version with in reason. Like upgrading from 1.13 to 1.19 would be expected to pose no real issues; however, jumping from 1.10 directly to 1.19 may pose some challenges. That being said, rolling back is not something that can be guaranteed to go smoothly as some versions can result in changes to database structure/content, modification/change to NiFi components (processor, controller services, reporting tasks, etc). In a best case scenario, a rollback my result in a working NiFi, with unexpected unexpected invalid components on the NiFi canvas. This happens when new properties are introduced in components being used, Those component changes get written to the flow.xml.gz and new flow.josn.gz files. So on rollback those new properties turn in to dynamic properties which may result in an invalid component until they are removed manually. In other scenarios, there may have been upgrades/changes to H2 DB that won't rollback smoothly. So perhaps taking a backup of the flow.xml.gz. flow.json.gz, and database_repository/ directory contents would be useful in the vent of rollback. Good news is that there is nothing in the H2 DBs located in the database_repository you need to retain. The nifi-flow-audit DB tracks history of changes you made on the canvas. The other user and identity dbs can just be removed and allowed to be recreated on next startup. So in your rollback test, I would rename the existing DB files and restart allowing new to be created. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
04-04-2023
12:13 PM
@davehkd Sounds like your NiFi-Registry service install went well. There is not such thing as a NiFi registry tool. I assume you were referring to the NiFi TLS toolkit you used to generate your certificates and keystores? You are dealing with an issue related to your initial admin. 1. When you configured the user-group-provider and file-access-policy provider, you stated that you used your "ec2-user" DN; however, the authorization exception you shared is as follows" AccessDeniedExceptionMapper: identity CN=ec2-user does not have permission to access the requested resource. That is not a the user's full DN in that exception. To NiFi and NiFi-Registry the user identity string "CN=ec2-user. OU=NiFi" is not the same user as "CN=ec2-user". What appears to me from your description is the full user DN has been created in the users.xml and authorization for that full DN setup in the authorizations.xml. So to answer why your "CN=ec2-user, OU=NiFi" DN resulted in NiFi-Registry seeing your user identity as only "CN=ec2-user" comes down to identity mapping properties. So in yoou nifi-registry.properties file you'll want to look for sets of properties like the following: nifi.registry.security.identity.mapping.pattern.<some string>=<some regex pattern with 1 or more capture groups>
nifi.registry.security.identity.mapping.value.<some string>=$1
nifi.registry.security.identity.mapping.transform.<some string>=NONE My guess here is you will find some regex pattern that matches on your user's full DN and the corresponding value property is then only returning "CN=ec2-user". You can remove or modify the matching mapping pattern or change your initial admin to just "CN=ec2-user" instead of using full DN. Either will work. Keep in mind if you decide to go with "CN=ec2-user", then you will need edit the authorizers.xml to use "CN=ec2-user" instead of full DN. You will also need to remove the current users.xml and authorizations.xml files created by the authorizer on first startup so they get recreated with these new identity strings. The authorizer providers only generate the authorizations.xml and users.xml on startup if they do NOT already exist. 2. the following output you reported from your nifi-registry-app.log is normal and expected: Kerberos service ticket not supported by the NiFi Registry. This is not an ERROR log. It is simply reporting that the spnego related kerberos properties in the nifi-registry.properties file were not configured and thus kerberos authentication via spnego is not supported. You can just ignore this notification. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
03-30-2023
02:44 PM
1 Kudo
@wffger2 @cotopaul The InstanceIdentifier was introduced when NiFi switched from using the flow.xml.gz to the flow.josn.gz for flow persistence. It is also currently being written to the flows stored in the NiFi-Registry. I say "currently" because this jira exist and is still open: https://issues.apache.org/jira/browse/NIFI-11321 So as @cotopaul points out, the "instanceIdentifier" will be unique for each instantiation of the flow definition, but the "identifier" will be the same. Downside here is every time you want to do a comparison, you'll need to download latest flow definition from DEV and UAT environments and compare for difference based on the Identifiers. Thanks, Matt
... View more
03-30-2023
01:37 PM
@nabilraza456 I am curious about the solution you said worked in HDF? Can you share that with me? I'd like to explore how that capability changed going to CFM. Thanks, Matt
... View more
03-30-2023
01:31 PM
@cotopaul Is that complete stack trace from the nifi-app.log? What version of Apache NiFi? What version of Java? Have you tried using ConsumeKafkaRecord processor instead of ConsumeKafka --> MergeContent? Do you have issue only when using the ParquetRecordSetWriter? How large are the FlowFiles coming out of the MergeContent processor? Have you tried reducing the size of the Content being output from MergeContent processor? Thanks, Matt
... View more
03-30-2023
01:13 PM
@saquibsk Stop your NiFi service. Try using the following command to reset the single user "username" and "password" to new values: $ ./bin/nifi.sh set-single-user-credentials <username> <password> Then start your NiFi service and see it you can successfully login. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more