@girish6    NiFi processor components are configured to execute based on a run schedule. There are two schedule driven strategies available (Cron Driven and Timer Driven). The Cron Driven scheduling strategy uses a user configured Quartz Cron to set how often the processor will execute.   The Timer Driven scheduling strategy (most common strategy used) uses a user configured run schedule (default run schedule is 0 secs, which means run as often as system will allow).   When a processor executes based on the configured scheduling strategy, it will do one of two things: 1. If the processor has one or more inbound connections, it will check if any of them have any queued FlowFiles.  If none of the connections contain any queued FlowFiles, the processor will yield.  The yield is intended to keep the processors with run schedule of 0 secs from simply constantly requesting CPU threads to check empty inbound connection queues.  No matter the run schedule, a yielded processor will not execute until the yield has expired reducing CPU usage by that processor.   2. Some processor have no inbound connections.  These processors will not yield, but continuously execute on the configured run schedule.  You would not have any such processors in your PG2 since they will have upstream connections to components in PG1.  So for "source" type processors like listSFTP, ListFile, GenerateFlowFIle, or any other processor that does not support an inbound/upstream connection,  if the feed of data is not continuous, it is best to use the Cron Driven scheduling strategy or set a Timer Driven run schedule that is not the default 0 secs to reduce CPU usage. On the face of every processor is a state for Tasks/Time.  The stat tells you how many threads reported as completed in the past 5 minutes and how much cumulative CPU time was used by all those completed threads.  This allows you to see the impact a given processor is having on your CPU. Hope this helps explain cpu usage for you, Matt   ... View more

@MattWho  Follow your points i got win. Thank you a lot. Paul  ... View more

@Matt Thanks, I solved this issue when i follow your point. Paul   ... View more

@Matt Thank you,  I'm doing what you point me to do.   ... View more

@Jette    There is no need for the updateAttribute processor here, unless there are some missing details to this issue. The extractText processor already permanently adds any created attribute to the FlowFile it outputs.  The dynamic property name used becomes the FlowFile attribute name.   Matt ... View more

hello, I see there is POST option to save a bundle/nar into the NiFi Registry however that can only happen from a local machine.   Is it possible to fetch bundle/nar from an artifactory and be able to save it in the NiFi Registry?  The reason I am thinking in that direction is because in our current SDLC process, Jenkins will do the build and resolve all dependencies and create the NAR and put it in the Artifactory.  How does the bundle gets deployed from the artifactory into the Registry, that is what I am after.    Thanks! ... View more

@Elephanta    Based on the information provided, here are some things to be aware of/consider:   How a Merge based processor decides to Merge a bin: 1. At end of thread execution a bin has reached both minimums (Min size and min records) 2. The max bin age of a bin has been reached 3. at start of a thread execution there are no free bins, forces oldest bin to merge to free a bin.   JVM heap memory: 1. While your system has 512GB of memory, how much of that has been allocated to NiFi's JVM.  Setting a very large heap for the JVM can result in significant stop-the-world application pauses even when minor Garbage Collection (GC) occurs.  Setting JVM heap too low when you have high heap usage processor in use can result in Out Of Memory (OOM) exceptions.  2. Merge based processors have potential for high heap usage.  While Merge content does not hold content of all FlowFiles being merged in heap memory, it does hold the FlowFile AttrIbutes of all binned FlowFiles in heap memory. So with a significant number of bins and large min record settings, this can cause high heap usage.  This intern can lead to excessive GC occurring. Processor configuration: 1. What is being used as your correlation attribute?  Are there more than 64 possible unique correlation attribute values?  This could lead to force merging of bins in mergeRecord processor 1-3. 2. With per bin record range set 100,000 - 10,000,000, you run the risk of high heap usage, excessive GC at times, or OOM.  Do expect that each unique correlation attribute will have this many records?  Perhaps a bin never meets your minimums and merge is only happening because of max bin age.  This would explain large pauses and small output FlowFiles. 3. Knowing your incoming data to a merge processor is critical when setting min and max values.  Since both mins must be satisfied, you can run in to s scenario where max records is reached, but you did not reach min bin size.  That result in bin being forced to sit until max bin age forces it to merge since both min values were not met and because one of the max values was met nothing additional could be allocated to that bin.  Again, this can explain your long pauses and small files sizes. 4. you did not mention if your NiFi is a cluster or standalone (single) NiFi instance installation.  If a cluster, keep in mind that each node can only merge FlowFIles which exist on that same node.  Nodes are not aware of FlowFiles on other nodes.  However, since you are merging based on a correlation attribute, you can configure a connection to load-balance data across all your nodes based on that same correlation attribute.  This would allow you to use parallel processing to merge your large bundles across multiple NiFi nodes. Threading: 1. When a processor executes, it must requests a thread from the NiFi core.  The core has a configurable Max Timer Driven Thread Pool (found in controller setting under the global menu in upper right corner).  By default this thread pool is only set to 10.  This thread pool is shared by all components you add to your canvas. With 128 cores, the recommended setting for the pool would be 256 - 512 (of course you must also take in to consideration what else may be running in this server, so monitor your cpu usage over time and adjust accordingly.)   Disk I/O: 1. NiFi writes all its data in to content claims on disk.  We strongly recommend that NiFi's content, flowfile, and provenance repositories are located on separate disks to improve IO and reduce likely hood of corruption of flowfile repo should content repo fill disk to 100%. 2. To help reduce heap usage of actively queued FlowFiles.  NiFi will begin writing swap files to disk when a connection queue exceeds the configured swap threshold set in the nifi.properties file.  (Note: the connection queue feeding your merge processor may or may not contained swapped FlowFiles. FlowFiles allocated to bins will still show in the connection but will not be eligible to be swapped to disk.)   Data ingestion: 1. Your source records seem very small. How is your data being ingested in to NiFi.  Perhaps a different method, or processor configuration can yield fewer yet large records.  This would result in more efficient merging and less disk swapping.   Here are some articles you may want to read: https://community.cloudera.com/t5/Community-Articles/HDF-NIFI-Best-practices-for-setting-up-a-high-performance/ta-p/244999 https://community.cloudera.com/t5/Community-Articles/Dissecting-the-NiFi-quot-connection-quot-Heap-usage-and/ta-p/248166 https://community.cloudera.com/t5/Community-Articles/Understanding-NiFi-max-thread-pools-and-processor-concurrent/ta-p/248920 https://community.cloudera.com/t5/Community-Articles/Understanding-NiFi-processor-s-quot-Run-Duration-quot/ta-p/248921   What you are trying to do is definitely doable with NiFi, but may require some dataflow design and/or system tuning to achieve. Hope this helps, Matt ... View more