@DayDream  The ExecuteStreamCommand processor executes a system level command and not something native within NiFi, so its impact on CPU is completely dependent on what the command being called is doing. You mention that the ExecuteStreamCommand is just executing a CP command and that issue happens when you are dealing with a large file.  The first thing I would be looking in to is disk I/O of the source and destination directory location where the file is being copied from and copied to. You also mention that the PutFile is writing out a large FlowFile to disk.  This means that the processors is reading FlowFile content from the NiFi content_repository and then writing it to some target folder location.  I would once again look at the disk I/O of both locations when this is happening.  The CPU usage may be high simply because these threads are running a long time waiting on disk I/O. NiFi uses CPU for its core level functions and then you configure an additional thread pool that is used by the NiFi components you add to the NiFi canvas.  This resource pool is configured via NiFi UI --> Global Menu (upper right corner of UI) --> Controller Settings: The "Event Driven" thread pool is experimental and deprecated and is used by processors configured to use the event driven scheduling strategy.  Stay away from this scheduling strategy. The "Timer Driven" thread pool is used by controller services, reporting tasks, processors, etc... The Processors will use it when configured to use the "Timer Driven" or "Cron driven" scheduling strategies. This pool is what is available for the NiFi controller to hand out to all processors requesting time to execute.   Setting this value to an arbitrarily high value will simply lead to many NiFi components getting threads to execute but then spending excessive time in CPU wait as the time on the limited cores is time sliced across all active threads.  The general rule of thumb here is to set the pool to 2 to 4 times the number of available core on a single NiFi host/node.  So for your 8 core server, you would want this between 16 and 32.  This does not mean you can't set this higher, but should only do this in smaller increments while monitoring CPU usage over extended period of time.  If you have 5 nodes, this setting is per node so you would have a thread pool of 16 - 32 on each NiFi host/node. Another thing you may want to start looking at is the GC stats for your JVM.  Is GC (young and old) running very often?  Is it taking a long tome to run?  All GC is a stop-the-world event, so the JVM simply is paused while this is going on which can also impact how long a thread is "running". You can get some interesting details about your running NiFi using the built in NiFi diagnostics tool. <path to NiFi>/bin/nifi.sh diagnostics --verbose <path/filename where output should be written> For a NiFi node to remain connected to it must be successful at sending a heartbeat to the elected cluster coordinator at least 1 out of 8 scheduled heartbeat intervals.  Let's say the heartbeat interval is configured in the nifi.properties file for 5 secs, then the elected CC must successfully process at least 1 heartbeat every 40 secs or that node would get disconnected for lack of heartbeat.  The node would initiate a reconnection once a heartbeat is received after having been disconnected for above reason.  Configuring a larger heartbeat interval will help avoid this disconnect/reconnect by allowing from time before heartbeat is considered lost.  This would allow more time if the node is going through a long GC pause or the CPU is so saturated it can't get a thread to create a heartbeat.   I also recommend reading through this community article: https://community.cloudera.com/t5/Community-Articles/HDF-CFM-NIFI-Best-practices-for-setting-up-a-high/ta-p/244999 If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more

@CodeLa @SAMSAL  I want to point out that tracking timestamps will not always guarantee NiFi will consume all files from the input file directory depending on how they are being placed in that directory. The ListFile processor looks at the last modified timestamp on the file.  It then lists all files since the last recorded timestamp stored in NiFi state manager from the previous processor execution.  On first run their will be no state and this everything currently is listed. Now consider the scenarios below which can affect above from listing all files: The mechanism that is writing the files to that inout directory is not updating the last modified timestamp on the file once it is done writing to it.  Let say we have file 1 that starts being written to as 12:00:01.000 and file 2 that starts being written as 12:00:01.300. File 2 completes first and is consumed by listFile and stored state is updated to reflect 12:00:01.300.  Now File 1 completes, but is never consumed by ListFile since its last modified timestamp is older than file 2. If you are in such a scenario, the ListFile offers a different "Listing Strategy" called "Tracking Entities" which tracks filenames as well in a cache service which allows it to still list files that may have an older timestamp. Another thing to consider is listFile may list the same file more than once. Consider this scenario: You tell NiFi ListFile to list files from directory /nifi/myfiles/.  The mechanism writing these files to the target directory does update the last modified timestamp as file is being written, but does not use a ".<filename>" (dot rename) approach to writing these files (means file is initially a hidden file until file write completes and then is renamed and made unhidden. Default listFile config ignores hidden files).  So when ListFile runs, it sees that file with newer last modified timestamp and lists it.  Then on next execution it sees same file again because its last modified timestamp is updated as file is still being written to. If you are in such a scenario, you would want to make use of the "Minimum File Age" property.  This property tells the listFile to ignore any files were the last modified time stamp when compared to current time is not at least that configured amount of time old (that means last modified timestamp has not changed for configured amount of time).  That configured time is arbitrary and what ever length is needed for you to be confident file write was complete.  Something else you need to consider depends on if both the following are true: 1. You are using a multi node NiFi cluster 2. The configured directory you are listing from is mounted to every node. Since every node in a NiFi cluster is executing the same dataflow, you want to avoid every node from listing the same files. IN this scenario you would change the "Execution" configuration from "All nodes" to "Primary" on the ListFile and change "input Directory location" from "local" to "remote".  Then you will want to set "load balance Strategy" to "Round Robin" on the connection between ListFile and FetchFile. NOTE: Never set the Execution on any processor that has an inbound connection to "Primary node".  ONLY processor with not inbound connection should be considered for this execution configuration. I know this is a lot to digest, but very important to be aware of to ensure success. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt  ... View more

@Ronman  I don't know anything about the cetic/helm-nifi image. I assume you are talking about this: https://github.com/cetic/helm-nifi I started parsing through what is in the above github and the authorizers.xml that is built looks poorly done. Can you share what is in your authorizers.xml file on your NiFi host? The image looks to create numerous providers that are not actually used from what I can tell. Looks like it creates: 1. file-user-group-provider 2. ldap-user-group-provider (only if ldap enabled) 3. composite-configurable-user-group-provider (Only if ldap enabled) 4. file-access-policy-provider. (Always points at file-user-user-group-provider which means 2 and 3 would never get used even if they were created) 5. managed-authorizer (points at file-access-policy-provider) 6. file-provider (only if LDAP enabled.   This is a legacy provider and not sure why anyone would still use it. It can reference and of the above user-group providers) So seeing what is actually written to that file might be helpful here. Also on startup the Authorizers.xml is responsible for seeding some initial polciy for the admin user in the users.xml and authorizations.xml files.  This would including the intial set of policies for the root PG.  This will not happen if upon first launch of NiFi there was not flow.xml.gz yet and thus no flow.xml.gz containing the root PG UUID yet. So you may want to rename your existing authorizations.xml file and restart your NiFi so that a new one is generated since you have a flow.xml.gz now and see if that gives you the policies you need to start editing the canvas. But even if above works, it still think you have an issue within your authorizers.xml files configuration. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more