Member since
07-30-2019
3387
Posts
1617
Kudos Received
999
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 125 | 11-05-2025 11:01 AM | |
| 375 | 10-20-2025 06:29 AM | |
| 515 | 10-10-2025 08:03 AM | |
| 357 | 10-08-2025 10:52 AM | |
| 394 | 10-08-2025 10:36 AM |
01-29-2025
06:14 AM
@mslnrd Authentication and authorization are two separate configurations. A user must successfully authenticate before any authorization is checked. So from your description, you are getting the NiFi login in window and successfully authenticating using your AD sAMAccountName and password. This means that the case sensitive username you entered at login is being passed on for authorization handled by the configured authorizer in the authorizers.xml. The authorizers.xml is easiest to read from the bottom up starting with the authorizer. Looking at what you shared, we see the "managed-authorizer" being used which has a dependency on the "file-access-policy-provider" (which persists all the configured authorizations in the authorizations.xml file). Now looking at the "file-access-policy-provider", we see it has a dependency on the "file-user-group-provider" for understanding what groups an authenticated user belongs to. If we then look at the "file-user-group-provider", it simply allows you to manually define new user identities and associated them with manually defined group identities. Which from you query sounds like what you have been doing thus far. We can also see that you have added the "ldap-user-group-provider" to the authorizers.xml; however, from reading the file as i described above we can see no path of reference from authorizer to this ldap-user-group-provider. That means the authorizers is not using any users and groups this provider may be returning. Now fixing this configuration issue has two possible paths. 1. You can reconfigure the "file-access-policy-provider" to use the "ldap-user-group-provider" 2. You can configure the "file-access-policy-provider" to use a "Composite-configurable-user-group-provider" (which can be configured to get group info form multiple user-group-providers). Note:
You'll need to use the "Composite-configurable-user-group-provider" if using the configurable file-user-group-provider as one of the providers. The file-user-group-provider can NOT be configured in the "Composite-user-group-provider" Option 2 allows more flexibility because you can authorize server client auth certificates which are not typically in AD/LDAP. Such as authorizing NiFi nodes to talk to one another in cluster or authorizing one NiFi to connect to another NiFi via NiFi Site-To-Site capability. With Option 2, you need to be aware that multiple user group providers can NOT return the same user or group identity string. Since you have already added your users and groups manually via the file-user-group-provider, NiFi will error on startup complaining that multiple providers have returned the same identity. So you will need to rename/remove the existing users.xml file and unset the "Initial User Identity 1" field in the file-user-group-provider only. On Startup, NiFI will pull in user and groups via your ldap-user-group-provider configuration and you will still have the option to manually define additional non AD/LDAP user and group identities if needed via the NiFi UI. An example authorizers.xml setup of what is described above is found here in the NiFi Admin Guide: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#composite-file-and-ldap-based-usersgroups Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-28-2025
06:05 AM
1 Kudo
@Shampy Apache NiFi does NOT support the older flow.xml.gz format. It can only load the newer flow.json.gz format. Your Apache NiFi 1.27 should be producing both the flow.xml.gz and flow.json.gz flow storage formats. You'll need to use the flow.json.gz format in your NiFi 2.x installation. Apache NiFi 1.x introduced the newer flow.json.gz flow storage format in Apache NiFi 1.16 and newer. In those newer versions of Apache NiFi 1.16+ will generate the newer flow.json.gz format and still maintain the older flow.xml.gz format. This positions you for upgrading to Apache NiFi 2.x. You'll now have the flow.json.gz needed to load in your 2.x version. The proper path to Apache NiFi 2.x is to first to upgrade to the latest Apache NiFi 1.x release. Before upgrading to Apache NiFi 2.x version, you should review the release notes between your current version to the version you plan to upgrade to. This allow you to see if you are using and components that have been removed or if any breaking changes impact your dataflows: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes/#ReleaseNotes-Version2.1.0 NOTE:
In Apache NiFi 1.x versions that support both the flow.xml.gz and the newer flow.json.gz format, the flow.xml.gz format file will be ignored on startup if a flow.json.gz exists. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-27-2025
11:06 AM
@vystar Considering the breaking changes that are part of Apache NiFi 2.0/1, there is considerably more work in preparing for an upgrade to the that new major release. So I would recommend upgrading to the latest offering in the Apache NiFi 1.x branch. You'll want to review all the release notes from 1.13 to the latest release: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes/#ReleaseNotes-Version1.12.0 You'll want to pay close attention to any mentions of components being moved to optional build profiles. This means that these nars and the components they contain are no longer include with the Apache NiFi download and if needed must be downloaded form Maven Central and manually added to NiFi. Deprecated components still exist in the Download, but will not exist in NiFi 2.x releases. Make sure to maintain a copy of your flow.xml.gz/flow.json.gz (newer releases). The newer Apache NiFi 1.x load a flow.json.gz instead of the older flow.xml.gz on startup. However, in the absence of a flow.json.gz and the presence of flow.xml.gz, NiFi 1.x will load from the flow.xml.gz and produce the new flow.json.gz. After upgrade, you'll still need to review your dataflows. There are some bad practices that are now blocked by Apache NiFi that may leave some components invalid until manual action is taken to resolve the bad configuration (such as using "primary node" execution on any processor that has an inbound connection). As far as FlowFile distribution, do it early in your dataflows as possible. Utilize list/fetch still processors instead of get style. (Example: use ListSFTP and FetchSFTP in place of GetSFTP. This allows you to load-balance the 0 byte listed files before the content is fetched for the files). Other options like Remote Process Groups can be used (they come with some overhead, but do some target NiFi Cluster load based distribution when dealing with large volumes of FlowFiles. Not so great for low volumes.). Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-27-2025
05:30 AM
@ose_gold Looping back on number 6. When you see the active thread indicated in the upper right corner, do you see tasks as completing or the thread just remains active all the time? Do you see any data being written to your Redis distributed map cache server? Thanks, Matt
... View more
01-27-2025
05:23 AM
@vystar Welcome to the community. The first observation is your NiFi version being 1.12.1 released 6 years ago. There have been a lot of bug fixes and improvements made to load balanced connection since then. I strongly encourage you to upgrade to much newer version of Apache NiFi. Once a NiFi connection has load balanced the FlowFiles in the connection, it will not redistribute them again. So if your other two nodes receive their round robin distribution and have capacity to process them faster the connection will not round robin the other FlowFiles in the connection left on 1 node again. Doing so would be very expensive as each node would be trying to redistribute already round robin distributed FlowFiles over and over again. Maximizing throughput in NiFi often requires looking at all your dataflows, configurations, designs, memory and cpu usage data. Is the ExecuteStreamCommand processor the only slow point in your dataflow? What is it executing? how is it configured? Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-24-2025
11:39 AM
1 Kudo
@ose_gold Welcome to the community. Can you share more details. Try using the latest Apache NiFi 2.1.0 release instead of the very first unofficial maintenance release 2.0.0. Does the processor show as valid? From the server running NiFi, can you resolve and connect to the FTP server via command line? Do any of the files on the FTP server have a last modified timestamp newer then 3 hours old? Try changing "Entity Tracking Initial Listing Target" from "tracking time window" to "All available". Does it produce a FlowFiles? When you start listFTP processor, does it indicate an active running thread in the upper right corner? What do you see in the nifi-app.log when processor runs? Any exceptions? Try puttingg this processor class (org.apache.nifi.processors.standard.ListenFTP) in DEBUG in the NiFi logback.xml file to get additional logging output. With using "tracking timestamps" instead does it produce flowfiles? Hopefully some of the above checks can help narrow focus of where the issue exists. Thanks you, Matt
... View more
01-24-2025
07:01 AM
@AbhiTryingAgain Welcome to the community and to Apache NiFi. Before building Dataflows it is important to understand the basics of a NiFi FlowFile. This will help you navigate the available processor components and the expectations on what they do at a high level. NiFi utilizes FlowFile so that it can remain data agnostic allowing NiFi to handle content of any type. Now performing actions against the content of a FlowFile would require processors that can understand the data format of the content. A NiFi FlowFile is what is transferred from one NiFi Processor on the canvas to the next. A FlowFile consists of two parts: FlowFile Content- This is the actual binary data/content. NiFi persist this content in content claims within the NiFi content_repository. FlowFile Attributes/Metadata - This is attributes and metadata about the content and FlowFile. At the most basic level, all FlowFiles will have timestamps, filename, etc attributes. Various NiFi processor will add even more attributes to a FlowFile. Understanding above, NiFi processors like "RouteOnAttribute" never look at the content of a FlowFile, it only looks at the FlowFile attributes of a FlowFile and route the FlowFile to the specified dynamic downstream relationship. So when you setup three routes, they all evaluated to 'true' and FlowFile was cloned and routed to all three downstream relationships. What you need is a NiFi processor that will evaluate the multiple json records in your FlowFile's content and output multiple FlowFiles based on a unique category value. For this, I think the PartitionRecord processor is whet you could use. This avoids splitting you Json record in to multiple records and then merging the various splits back into single FlowFiles based on category. You can then use the JsonTreeReader and JsonRecordSetWriters. Based on your example, configurations would look like this: Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-23-2025
10:27 AM
@spartakus You will want to review the release notes for all release between 1.23. and 1.28.1 https://cwiki.apache.org/confluence/display/NIFI/Release+Notes/#ReleaseNotes-Version1.23.0 You'll notice a number of components deprecated. Deprecated components are not removed, but that are marked as deprecated and will log if used that they are deprecated. Deprecated components will eventually be removed form Apache NiFi (most removed in NiFi 2.0). You'll also make note of any items in the release notes that state: Moved <XYZ> to optional build profiles This means that these components were removed from Apache NiFi. These components can be added manually later by downloading the nars from the central maven repository and adding them to the NiFi lib directory if you need them still. https://mvnrepository.com/artifact/org.apache.nifi Make a back up of your flow.json.gz, nifi.properties, and optionally flow.xml.gz files before you upgrade. Otherwise, I see no issues with upgrading from 1.22 -- > 1.28 directly. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-22-2025
09:13 AM
@amitjha Providing the full stack trace from the nifi-app.log may help here in identifying the issue. Is the controller service enabled and valid? any other exceptions in the logs related to either the consumeJMS processor or the JMSConnectionFactoryProvider controller service? The connectionFactory will on attempt to create the connection once first referencing component request a connection. The exception eludes to and issue there, but full stack trace may help identify exactly what went wrong. Have you tried this tutorial from Solace: https://solace.com/integration-hub/nifi/ Other observations: Keep in mind that you are using an HDF release that is going on 8 years old. I recommend not adding outside jars directly to the NiFi lib directory. Create a directory outside of NiFi's base directory and add your custom jars there. Just make sure the NiFi service user can access that directory and owns the jars. Thanks, Matt
... View more
01-22-2025
08:42 AM
@anon12345 Yes, you need to unset the keystore and truststore properties because NiFi will still attempt to load them even if HTTPS properties are unset. This is by design because NiFi can act as a client and as a client it may attempt to connect to secured endpoints where NiFi would use this keystore and trustore. So ay property configured in nifi.properties will be loaded, so they need to be unset or have valid values set. For example: NiFi's Site-To-Site capability. A unsecure NiFi configured with a valid keystore and truststore can have a Remote Process Group configured to connect to a Remote port on another secured NiFi. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more