Member since
07-30-2019
3172
Posts
1571
Kudos Received
918
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 96 | 02-14-2025 08:43 AM | |
| 101 | 02-12-2025 10:34 AM | |
| 467 | 02-06-2025 10:06 AM | |
| 143 | 01-31-2025 09:38 AM | |
| 127 | 01-30-2025 06:29 AM |
01-24-2025
11:39 AM
1 Kudo
@ose_gold Welcome to the community. Can you share more details. Try using the latest Apache NiFi 2.1.0 release instead of the very first unofficial maintenance release 2.0.0. Does the processor show as valid? From the server running NiFi, can you resolve and connect to the FTP server via command line? Do any of the files on the FTP server have a last modified timestamp newer then 3 hours old? Try changing "Entity Tracking Initial Listing Target" from "tracking time window" to "All available". Does it produce a FlowFiles? When you start listFTP processor, does it indicate an active running thread in the upper right corner? What do you see in the nifi-app.log when processor runs? Any exceptions? Try puttingg this processor class (org.apache.nifi.processors.standard.ListenFTP) in DEBUG in the NiFi logback.xml file to get additional logging output. With using "tracking timestamps" instead does it produce flowfiles? Hopefully some of the above checks can help narrow focus of where the issue exists. Thanks you, Matt
... View more
01-24-2025
07:01 AM
@AbhiTryingAgain Welcome to the community and to Apache NiFi. Before building Dataflows it is important to understand the basics of a NiFi FlowFile. This will help you navigate the available processor components and the expectations on what they do at a high level. NiFi utilizes FlowFile so that it can remain data agnostic allowing NiFi to handle content of any type. Now performing actions against the content of a FlowFile would require processors that can understand the data format of the content. A NiFi FlowFile is what is transferred from one NiFi Processor on the canvas to the next. A FlowFile consists of two parts: FlowFile Content- This is the actual binary data/content. NiFi persist this content in content claims within the NiFi content_repository. FlowFile Attributes/Metadata - This is attributes and metadata about the content and FlowFile. At the most basic level, all FlowFiles will have timestamps, filename, etc attributes. Various NiFi processor will add even more attributes to a FlowFile. Understanding above, NiFi processors like "RouteOnAttribute" never look at the content of a FlowFile, it only looks at the FlowFile attributes of a FlowFile and route the FlowFile to the specified dynamic downstream relationship. So when you setup three routes, they all evaluated to 'true' and FlowFile was cloned and routed to all three downstream relationships. What you need is a NiFi processor that will evaluate the multiple json records in your FlowFile's content and output multiple FlowFiles based on a unique category value. For this, I think the PartitionRecord processor is whet you could use. This avoids splitting you Json record in to multiple records and then merging the various splits back into single FlowFiles based on category. You can then use the JsonTreeReader and JsonRecordSetWriters. Based on your example, configurations would look like this: Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-23-2025
10:27 AM
@spartakus You will want to review the release notes for all release between 1.23. and 1.28.1 https://cwiki.apache.org/confluence/display/NIFI/Release+Notes/#ReleaseNotes-Version1.23.0 You'll notice a number of components deprecated. Deprecated components are not removed, but that are marked as deprecated and will log if used that they are deprecated. Deprecated components will eventually be removed form Apache NiFi (most removed in NiFi 2.0). You'll also make note of any items in the release notes that state: Moved <XYZ> to optional build profiles This means that these components were removed from Apache NiFi. These components can be added manually later by downloading the nars from the central maven repository and adding them to the NiFi lib directory if you need them still. https://mvnrepository.com/artifact/org.apache.nifi Make a back up of your flow.json.gz, nifi.properties, and optionally flow.xml.gz files before you upgrade. Otherwise, I see no issues with upgrading from 1.22 -- > 1.28 directly. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-22-2025
09:13 AM
@amitjha Providing the full stack trace from the nifi-app.log may help here in identifying the issue. Is the controller service enabled and valid? any other exceptions in the logs related to either the consumeJMS processor or the JMSConnectionFactoryProvider controller service? The connectionFactory will on attempt to create the connection once first referencing component request a connection. The exception eludes to and issue there, but full stack trace may help identify exactly what went wrong. Have you tried this tutorial from Solace: https://solace.com/integration-hub/nifi/ Other observations: Keep in mind that you are using an HDF release that is going on 8 years old. I recommend not adding outside jars directly to the NiFi lib directory. Create a directory outside of NiFi's base directory and add your custom jars there. Just make sure the NiFi service user can access that directory and owns the jars. Thanks, Matt
... View more
01-22-2025
08:42 AM
@anon12345 Yes, you need to unset the keystore and truststore properties because NiFi will still attempt to load them even if HTTPS properties are unset. This is by design because NiFi can act as a client and as a client it may attempt to connect to secured endpoints where NiFi would use this keystore and trustore. So ay property configured in nifi.properties will be loaded, so they need to be unset or have valid values set. For example: NiFi's Site-To-Site capability. A unsecure NiFi configured with a valid keystore and truststore can have a Remote Process Group configured to connect to a Remote port on another secured NiFi. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-21-2025
06:16 AM
@jirungaray Cloudera Flow Management (Based on Apache NiFi) provides multiple methods for managing user authorization. This includes NiFi internally via the File-Access-Policy-Provider and externally via Apache Ranger. There is no built in mechanism for auto setting up authorization policies for users or groups with the exception of the Initial Admin and Initial NiFi Node authorizations. Many of the Authorization policies are directly related to the components added to the canvas. Those components are assigned unique IDs making it impossible to create policies before the components exist. File-Access-Policy-Porvider: This provider utilizes a file on disk (authorizations.xml) to persists authorization policies. This file is loaded when NiFi starts. This means it is possible to manually generate this file and have NiFi load it on startup. Also as you mentioned, you could script out the authorization creating through NiFi Rest-API calls. Ranger provider: This moves authorization responsibility over to Apache Ranger. Policies setup within Ranger are download by the NiFi nodes where they are locally enforced. No matter which authorizer you choose to use, authorizations are easiest to manage via groups. Typical users setup ldap groups for various NiFi roles (admins, team 1, team2, etc..) and makes specific users members of these groups. This simplifies authorization since you can authorizer these groups instead of the individual users. Simply adding or removing a user as member of one of these authorized groups gives or removes authorized access to the NiFi resource identifier (NiFi policy). The ldap-user-group-provider can be added to the NiFi authorizers.xml to auto manage syncing of user and group identities from your AD/LDAP further simplifying management over the file-user-group-provider method which requires the manual adding of user and group identifiers to the NiFi. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-16-2025
03:22 PM
@tono425 I assumed all your records were of the same schema. With Merge Record, a bin will consist of potentially many 'like FlowFiles'. In order for two FlowFiles to be considered 'like FlowFiles', they must have the same Schema (as identified by the Record Reader). If a FlowFile is not like other FlowFiles already allocated to a bin, it will be allocated to a different bin. I would still recommend against using min record to 1 since a typical dataflow will be a constant stream of new FlowFiles and the MergeRecord processor will only see those FlowFiles queued at the exact moment of execution. So you can result in smaller then expected number of records in a merge Record with a constantly running dataflow. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-16-2025
05:41 AM
@tono425 When the mergeRecord processor executes it allocates FlowFiles from the inbound connection to bins. at the end of that execution it determines if any of the bins are eligible to be merged. Since you have Minimum number of records set to 1, a bin would merge even if it only had 1 record in it. Understand that Merge processor will not wait for the max setting to be reached. Try setting your "Min num of records" to 20000 and set your "max bin age" to some value like 5 minutes (The max bin age controls how long Merge record will wait for a bin to reach the set mins before forcing the merge with fewer records when using Bin-Packing Algorithm) Also be mindful of the number of FlowFile it takes to make up the 20000 records you are trying to merge since a FlowFile can contain 1 too many records in it. Also keep in mind that if you are running a NiFi Cluster setup, each node can only merge FlowFiles located on the same node. The Merge processor will not merge FlowFiles across nodes. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-16-2025
05:31 AM
@Eslam Welcome to the community. In order to get helpful answers, you'll need to provide more detail around your use case. NiFi provides many processors for connecting to various services on external hosts. You can find the list of default processors available with the Apache NiFi release here: NiFi 1.x release: https://nifi.apache.org/docs/nifi-docs/ NiFi 2.x release: https://nifi.apache.org/components/ At the very basic of level you have processors like: GetSFTP ListSFTP / FetchSFTP But there are also processor for connecting SMB, Splunk, rest-apis, SNMP, FTP, DBs, Kafka, hive, etc. on external servers. Look through the components list in the documentation for Get, List, Fetch, and Query type processors to see if any of them meet your use case needs. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-15-2025
06:25 AM
@itninja Did you configure your NiFi to use HTTPS or is still configured for HTTP? In order for NiFi to support user authentication and authorization it must be secured over HTTPS. Could you share some screenshots so I better understand what you mean by: Manage to create Administrator user with the set user script. Able to login with Administrator. But the anonymous is still enabled. Code created user Administrator is still visible with anonymous user. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more