Member since
07-30-2019
3473
Posts
1642
Kudos Received
1021
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 287 | 06-15-2026 08:08 AM | |
| 492 | 06-03-2026 06:06 PM | |
| 728 | 05-06-2026 09:16 AM | |
| 1682 | 05-04-2026 05:20 AM | |
| 818 | 05-01-2026 10:15 AM |
10-19-2021
12:04 PM
@DayDream The ExecuteStreamCommand processor executes a system level command and not something native within NiFi, so its impact on CPU is completely dependent on what the command being called is doing. You mention that the ExecuteStreamCommand is just executing a CP command and that issue happens when you are dealing with a large file. The first thing I would be looking in to is disk I/O of the source and destination directory location where the file is being copied from and copied to. You also mention that the PutFile is writing out a large FlowFile to disk. This means that the processors is reading FlowFile content from the NiFi content_repository and then writing it to some target folder location. I would once again look at the disk I/O of both locations when this is happening. The CPU usage may be high simply because these threads are running a long time waiting on disk I/O. NiFi uses CPU for its core level functions and then you configure an additional thread pool that is used by the NiFi components you add to the NiFi canvas. This resource pool is configured via NiFi UI --> Global Menu (upper right corner of UI) --> Controller Settings: The "Event Driven" thread pool is experimental and deprecated and is used by processors configured to use the event driven scheduling strategy. Stay away from this scheduling strategy. The "Timer Driven" thread pool is used by controller services, reporting tasks, processors, etc... The Processors will use it when configured to use the "Timer Driven" or "Cron driven" scheduling strategies. This pool is what is available for the NiFi controller to hand out to all processors requesting time to execute. Setting this value to an arbitrarily high value will simply lead to many NiFi components getting threads to execute but then spending excessive time in CPU wait as the time on the limited cores is time sliced across all active threads. The general rule of thumb here is to set the pool to 2 to 4 times the number of available core on a single NiFi host/node. So for your 8 core server, you would want this between 16 and 32. This does not mean you can't set this higher, but should only do this in smaller increments while monitoring CPU usage over extended period of time. If you have 5 nodes, this setting is per node so you would have a thread pool of 16 - 32 on each NiFi host/node. Another thing you may want to start looking at is the GC stats for your JVM. Is GC (young and old) running very often? Is it taking a long tome to run? All GC is a stop-the-world event, so the JVM simply is paused while this is going on which can also impact how long a thread is "running". You can get some interesting details about your running NiFi using the built in NiFi diagnostics tool. <path to NiFi>/bin/nifi.sh diagnostics --verbose <path/filename where output should be written> For a NiFi node to remain connected to it must be successful at sending a heartbeat to the elected cluster coordinator at least 1 out of 8 scheduled heartbeat intervals. Let's say the heartbeat interval is configured in the nifi.properties file for 5 secs, then the elected CC must successfully process at least 1 heartbeat every 40 secs or that node would get disconnected for lack of heartbeat. The node would initiate a reconnection once a heartbeat is received after having been disconnected for above reason. Configuring a larger heartbeat interval will help avoid this disconnect/reconnect by allowing from time before heartbeat is considered lost. This would allow more time if the node is going through a long GC pause or the CPU is so saturated it can't get a thread to create a heartbeat. I also recommend reading through this community article: https://community.cloudera.com/t5/Community-Articles/HDF-CFM-NIFI-Best-practices-for-setting-up-a-high/ta-p/244999 If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
10-19-2021
11:32 AM
@vikrant_kumar24 The ExecuteScript processor has been around for over 6 years as part of Apache NiFi. It has had many improvements and bug fixes over those years just like many other well used components. I'd be reluctant from calling it "experimental" any longer regardless of what the embedded Apache NiFi docs say. The only thing to note here is that the ExecuteScript processor does not really execute the "Python" script engine. It is executing "Jython" instead which is a Java implementation of Python. Jython is not 100% compatible with Python, so you must test you script thoroughly. Thanks, Matt
... View more
10-08-2021
11:06 AM
@Ankit13 How do you know no more files we will be put after the NiFi flow processing starts? To me in sound like the PutFile should execute at default 0 secs (as fast at it can run) and you should instead control this dataflow at the beginning were you consume the data. For example: In a 24 hour window data is being written to source directory to be consumed from between 00:00:00 and 16:00:00. Then you want to write that data to target directory starting at 17:00. So you instead setup a cron on a listFile processor to consume list the files at 17:00 and 17:01 and then have a FetchFile and PutFile running all the time so these immediately consume all the content for the listed files and write them to target directory. Then your listFile does not execute again until same time next day or whatever you cron is. This way the files are all listed at same time and the putFile can execute for as long as needed to write all those files to the target directory. Hope this helps, Matt
... View more
10-07-2021
02:40 PM
@CodeLa @SAMSAL I want to point out that tracking timestamps will not always guarantee NiFi will consume all files from the input file directory depending on how they are being placed in that directory. The ListFile processor looks at the last modified timestamp on the file. It then lists all files since the last recorded timestamp stored in NiFi state manager from the previous processor execution. On first run their will be no state and this everything currently is listed. Now consider the scenarios below which can affect above from listing all files: The mechanism that is writing the files to that inout directory is not updating the last modified timestamp on the file once it is done writing to it. Let say we have file 1 that starts being written to as 12:00:01.000 and file 2 that starts being written as 12:00:01.300. File 2 completes first and is consumed by listFile and stored state is updated to reflect 12:00:01.300. Now File 1 completes, but is never consumed by ListFile since its last modified timestamp is older than file 2. If you are in such a scenario, the ListFile offers a different "Listing Strategy" called "Tracking Entities" which tracks filenames as well in a cache service which allows it to still list files that may have an older timestamp. Another thing to consider is listFile may list the same file more than once. Consider this scenario: You tell NiFi ListFile to list files from directory /nifi/myfiles/. The mechanism writing these files to the target directory does update the last modified timestamp as file is being written, but does not use a ".<filename>" (dot rename) approach to writing these files (means file is initially a hidden file until file write completes and then is renamed and made unhidden. Default listFile config ignores hidden files). So when ListFile runs, it sees that file with newer last modified timestamp and lists it. Then on next execution it sees same file again because its last modified timestamp is updated as file is still being written to. If you are in such a scenario, you would want to make use of the "Minimum File Age" property. This property tells the listFile to ignore any files were the last modified time stamp when compared to current time is not at least that configured amount of time old (that means last modified timestamp has not changed for configured amount of time). That configured time is arbitrary and what ever length is needed for you to be confident file write was complete. Something else you need to consider depends on if both the following are true: 1. You are using a multi node NiFi cluster 2. The configured directory you are listing from is mounted to every node. Since every node in a NiFi cluster is executing the same dataflow, you want to avoid every node from listing the same files. IN this scenario you would change the "Execution" configuration from "All nodes" to "Primary" on the ListFile and change "input Directory location" from "local" to "remote". Then you will want to set "load balance Strategy" to "Round Robin" on the connection between ListFile and FetchFile. NOTE: Never set the Execution on any processor that has an inbound connection to "Primary node". ONLY processor with not inbound connection should be considered for this execution configuration. I know this is a lot to digest, but very important to be aware of to ensure success. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
10-07-2021
02:06 PM
@Ronman I don't know anything about the cetic/helm-nifi image. I assume you are talking about this: https://github.com/cetic/helm-nifi I started parsing through what is in the above github and the authorizers.xml that is built looks poorly done. Can you share what is in your authorizers.xml file on your NiFi host? The image looks to create numerous providers that are not actually used from what I can tell. Looks like it creates: 1. file-user-group-provider 2. ldap-user-group-provider (only if ldap enabled) 3. composite-configurable-user-group-provider (Only if ldap enabled) 4. file-access-policy-provider. (Always points at file-user-user-group-provider which means 2 and 3 would never get used even if they were created) 5. managed-authorizer (points at file-access-policy-provider) 6. file-provider (only if LDAP enabled. This is a legacy provider and not sure why anyone would still use it. It can reference and of the above user-group providers) So seeing what is actually written to that file might be helpful here. Also on startup the Authorizers.xml is responsible for seeding some initial polciy for the admin user in the users.xml and authorizations.xml files. This would including the intial set of policies for the root PG. This will not happen if upon first launch of NiFi there was not flow.xml.gz yet and thus no flow.xml.gz containing the root PG UUID yet. So you may want to rename your existing authorizations.xml file and restart your NiFi so that a new one is generated since you have a flow.xml.gz now and see if that gives you the policies you need to start editing the canvas. But even if above works, it still think you have an issue within your authorizers.xml files configuration. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
10-07-2021
05:54 AM
@Ronman Please share the version of Apache NiFi you have installed. Thanks, Matt
... View more
10-07-2021
05:47 AM
@Ankit13 Perhaps I don't understand your use case. Are you saying you have a NiFi dataflow that slowly ingested Data producing FlowFiles that work their way through your dataflow to this putFile processor? Then you want these 1000s of FlowFiles to queue up so that they can all be put to the local file system directory at the same time? So what is being suggested by @m_adeel is to use the NIPYAPI to automated the starting and stopping of the putFile processor at a given time. You could also do the same through NiFi REST_API calls. You would still have the challenge of when to stop it. Does the source of data ever stop coming in? Would you be able to put all the FlowFiles from the inbound connection queue to disk before more source FlowFiles started flowing in to the queue? Why the need to do this at a specific data and time? Thanks, Matt
... View more
10-06-2021
10:48 AM
@Ronman It is interesting that you are seeing a 403 in response to trying to add a new policy on the root Process Group (PG). By granting your user string (CN=admin, OU=NIFI) to the following global policies: you should have ability to set any additional policies anywhere. From the root canvas you can either right click and select "manage access policies" or click on the small key icon in the "Operate" panel that shows the PG name and PG assigned uuid to access the PG level access policies. First you would select the policy you want to add user or group strings to from the pull-down menu. If the policy has not yet had any users assigned to it, you'll need to click on "add policy" before being able to click on the Then click on the small "add users/groups to this policy" icon to the far right. Note: Child PGs added to the canvas will by default inherit all their policies from the parent PG. You will be given option to "override" this and set child specific users/groups on that child PG. You are stating that you encounter a 403 when you click on the "add" button? What do you see in the nifi-user.log when you perform that action? If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
10-06-2021
06:00 AM
@CodeLa Everything you do via the NiFi canvas when building out your dataflows is preserved in the flow.xml.gz file written to disk. On NiFi service start, the flow.xml.gz is loaded in to memory and and then FlowFiles from the the flowfile_repository are loaded back in to the dataflow connections they were last in prior to NiFi being stopped. In order for your your canvas to be blank on NiFi service startup, that flow.xml.gz is missing. By default NiFi creates an archived copy of the flow.xml.gz each time a change is made. The following properties from the nifi.properties file control where the flow.xml.gz is written and if/how the archiving is setup: Recovering the flow from archive is as simple as copying the latest "<timesatmp>_flow.xml.gz" from the configured archive directory to the configured NiFi configuration file location and renaming it to "flow.xml.gz". Make sure proper ownership and permissions are set after you copy and rename (Must be owned and accessible by the NiFi service user). As a future suggestion: Another way to protect against a total loss of your flow is running a NiFi cluster instead of a single NiFi instance. In a NiFi cluster, every NiFi node preserves its own identical copy of the flow.xml.gz file. So should catastrophe strike on one of the nodes, you can simply copy the flow.xml.gz from any one of the other nodes to recover the bad node. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
10-06-2021
05:46 AM
@Ankit13 NiFi processors support "Timer Driven" and "Cron Driven" Scheduling Strategies. Timer driven strategy allows you to specify a time interval for scheduling (for example: "30 secs" which means processor will execute every 30 seconds.). Cron Driven strategy supports a Quartz Cron [1] being used to specify when the processor should execute. There is a third option on some processors which is Event Driven that should not be used. It was created long ago and considered experimental. It is has since been deprecated due to improvement made in the Timer Driven strategy. It only remains in NiFi to avoid breaking flows of those who use it when they upgrade. Important things to understand about your ask: Let's assume you configure your PutFile to execute using the Cron Driven scheduling strategy and the inbound connection to the putFile processor has multiple FlowFiles queued. When the processor executes it will process only 1 of those FlowFiles from that inbound connection queue with default settings. The next queued FlowFile would not get processed until the next scheduled cron execution. While there is no way to make sure that every queued FlowFile is processed in in a single cron execution you can change the configured Run Duration: The Run Duration tells the processor to continue to use the same execution thread to execute against as many queued FlowFile as possible within the configured run duration time. Let say it takes more than 2 secs to write the very first FlowFile to the target directory. In that case, only one FlowFile would be processed. So there would be no perceived difference between a run duration of 0ms and 2s. In a NiFi cluster, each node in the cluster executes the dataflow against the FlowFiles queued on that same node. So if FlowFiles were queued on the inbound connection to PutFile on all nodes, each node would execute 1 at each cron interval processing through FlowFile(s) per node as described above. [1] https://community.cloudera.com/t5/forums/replypage/board-id/Questions/message-id/229905 If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more