@drewski7  New processors are created within the community all the time and the documentation for processors should include resource considerations for CPU usage and Memory usage.  Just because a processor list memory as a resource consideration, that impact is often a byproduct of how that processor has been configured. you can refer to the imbedded documentation in your installed NiFi instance or you can right click on a processor added to the canvas and select "view usage" from the displayed context menu to go directly to that components embedded documentation page. But processors like ReplaceText, SplitText, SplitContent, SplitJson,... would be examples. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@Bridewin  I two things you may want to try.... 1. GetFile processor was deprecated in favor of the newer ListFile --> FetchFile processors.  I'd recommend switching to these processors and see if you have the same observations. 2. I'd suggest enabling debug logging for the GetFile processor class to see what additional logging may show.  To do this, you would modify the logback.xml file in NiFi's conf directory. Add the below line down in this file where you see similar lines already. <logger name="org.apache.nifi.processors.standard.GetFile" level="DEBUG"/>    If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@Jagapriyan I suspect issue with last modified timestamps since missed files have older last modified timestamp than what was already consumed from the target directory that is compounded by the sub-directory structure.  My recommendation is  switch to using the listing strategy "Tracking Entities" instead.  Tracking Entities will keep track of filenames and timestamps so even an older timestamped file will get consumed if its filename is not in the tracked entities list stored in the distributed cache. Let me know is making this change resolves yoru issue. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@PepeClaro  While NiFi supports parallel thread execution, there is no way to guarantee that two threads execute at the exact same time.  So one NiFi component processor is unaware of what another NiFi Component processor is doing or when it is executing.  Processors that have an inbound connection on them use an inbound connection queued FlowFile as the trigger to perform start execution. Step 1 is to identify what NiFi component processors can be used to perform/execute your 3 processes: https://nifi.apache.org/docs.html I have no idea form your description what your 3 processes do, so I can't make any recommendations on what you can/should use. Step 2 is deciding how to interconnect these NiFi processor component and preserve data needed for downstream dataflow processing in your third process.  When a processor executes the response/return from the execution can result in modification to existing NiFi FlowFile's content, Creation of new FlowFile content, Creation of an entirely new FlowFile, Creation of new FlowFile attributes (key/value pairs), modification of FlowFile attributes, or none of the above depending on the NiFi component processor being used.  Since you mention that first 2 processes get info that is needed by process 3, so would need to take that into consideration for process 3. Where is that info go ing to end up (FlowFile Content or FlowFile attributes)?  How large is that info returned (does it make sense to put it in to attribute)?  Does that returned info need to be modified in any way before process 3? In your Flow as described, you have two Process Groups (PG),  Each of these PGs performs your process 1 and process 2.  Each will be executing independent of the other and thus can not guarantee execution at the exact same time.  Cron scheduling of a processor can give a better chance of same time execution but still not a guarantee since it only schedules when to request an available thread from the NiFi Max Timer Driven Thread pool.  If at time of request all threads are in use, it will execute as soon as thread becomes available. Now out of these two PGs you will have two FlowFiles that your third process depends on.  There is no way to tell a NiFi processor component to pull attributes or content from two different FlowFiles source FlowFiles.  So before process 3 you need to combine any needed attributes and or content from the two original FlowFiles into one FlowFile that Process 3 can use.  Hard to make a recommendation here since I don't know any details about your 3 processes, what the FlowFiles that are produced by Process 1 and 2 contain in terms of content and attributes, and what content and/or attributes from process 1 and 2 are needed by process 3.   I made suggestion about maybe being able to use the "defragment" merge strategy from the MergeContent processor to combine the FlowFiles from process 1 and process 2, but not enough detail to say or say without needing to do other modification before MergeContent.  To "defragment" (combine process 1 fragment with process 2 fragment), the FlowFiles produced by both process 1 and process 2 would need to have the following FlowFile attributes present and set correctly on each:   Name Description fragment.identifier Applicable only if the <Merge Strategy> property is set to Defragment. All FlowFiles with the same value for this attribute will be bundled together. fragment.index Applicable only if the <Merge Strategy> property is set to Defragment. This attribute indicates the order in which the fragments should be assembled. This attribute must be present on all FlowFiles when using the Defragment Merge Strategy and must be a unique (i.e., unique across all FlowFiles that have the same value for the "fragment.identifier" attribute) integer between 0 and the value of the fragment.count attribute. If two or more FlowFiles have the same value for the "fragment.identifier" attribute and the same value for the "fragment.index" attribute, the first FlowFile processed will be accepted and subsequent FlowFiles will not be accepted into the Bin. fragment.count Applicable only if the <Merge Strategy> property is set to Defragment. This attribute must be present on all FlowFiles with the same value for the fragment.identifier attribute. All FlowFiles in the same bundle must have the same value for this attribute. The value of this attribute indicates how many FlowFiles should be expected in the given bundle. segment.original.filename Applicable only if the <Merge Strategy> property is set to Defragment. This attribute must be present on all FlowFiles with the same value for the fragment.identifier attribute. All FlowFiles in the same bundle must have the same value for this attribute. The value of this attribute will be used for the filename of the completed merged FlowFile. Fragment.identifier, fragment.count, and segment.original.filename need to have same values on both FlowFiles.  Fragment.index would be unique. The result would be one output FlowFile with the FlowFile content of both original process 1 and process 2 FlowFiles which process 3 could the use.  Or if process 1 and 2 produce FlowFiles with just FlowFile Attributes you need and not content, you could set " Keep All Unique Attributes" as the attribute strategy so that the 1 merged FlowFile has all unique attributes form both source FlowFiles for process 3 to use. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt   ... View more

@PepeClaro  Your description is vague which makes it difficult to provide suggestions around incorporating them into a dataflow design. - What are these three "processes"? - How are those processes being executed? What processors in use for these 3 processes? - Are there any dependencies between these processes other then order of execution?  For example, is output from processes 1 and/or 2 needed by process 3?   - Do processes 1 and 2 need to be executed in parallel? - Is your NiFi a multi-node cluster?  - What are the triggers for these processes? Does it require a NiFi FlowFile to trigger each processes? What kicks off this entire process dataflow? The more detail the better would be helpful.  You may be able to set a fragment identifier, fragment count (2), and fragment index (1 or 2) for the first two process FlowFiles and then merge those fragments into one FlowFile that can trigger the third process. If either fragment is missing it will not merge and thus not trigger the 3 process. If not needing process 1 and 2 in parallel, then a single dataflows process1 --> process 2 --> process 3 where a failure anywhere along the dataflow prevents execution of next process. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@D5ha  Sometimes it is useful to know more about your environment to include the full NiFi version and java versions.   Since it is reporting issues as loading the flow: java.lang.Exception: Unable to load flow due to: java.util.zip.ZipException: invalid stored block lengths at org.apache.nifi.web.server.JettyServer.start  I would lean towards some issue/corruption of the flow.xml.gz and/or flow.json.gz on this node.  Since all nodes run the same exact copy of these files, I'd copy them from a good node to the node failing to start.  Depending on your NiFi version you may not have a flow.json.gz file (This format was introduced in the most recent versions). If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@Jagapriyan  As a daily job, i may suggest you tackle this differently. You know your source files are written between 8am - 9am each day.  So i would configure your listSFTP to run on a cron schedule so it runs every second from 9am-10am to make sure all files are listed.  Then knowing that your files may number 90+ (unknown on max) , I would configure your "Min Num of Entries" to some value you know the count will never reach.  Make sure "Max Num Entries" is set to a value higher than that.  Then configure the "Max Bin Age" to some time 30 mins?  What this does is allow MergeContent to continue to allocated FlowFiles to a bin for 30 minutes at which time the bin is forced to merge even if the min value has not be reached.  Doing this makes sure you get only one FlowFile out per bin per node.  That single FlowFile can then be used to trigger your putEmail used for notification.  Additionally, the merged FlowFile will have an attribute " merge.count" added that you can use in your email body to report number of FlowFiles that were ingested. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@RRosa  I am not clear on what you mean by " migrating the flow files"?  A NiFi FlowFile is the object that is traversed via connections between NiFi component processors on the NiFi canvas.   Are you talking about migrating your actively queued FlowFiles from NiFi cluster 1 (Apache NiFi 1.12.1) to NiFi cluster 2 (Apache NiFi 1.17.0)?    Or are you talking about migrating the flow.xml.gz file (contains everything you have configured on the canvas of yoru NiFi) from old cluster to new? General guidance for upgrading Apache NiFi can be found in the admin guide here: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#upgrading_nifi The only thing I see NOT covered in that guidance is the preservation of component state.  Within a cluster Component state may be stored depending on component in either a local state directory on each node (each node holds only state for that node) or stored in cluster state (written to ZK and shared across all nodes).   Now if you are installing the new version of NiFi on the same hosts where the old NiFi nodes were running, simply preserve the state configuration and the new nodes when started with a copy of the flow.xml.gz will continue to read and use same state.  Same goes for new nodes using the same external ZK that the previous nodes used (stop old before starting new hosts).   While the Documentation recommends that you process out all queued FlowFiles from cluster 1 before starting cluster 2 that is not required.  If new nodes point to same content, flowfile, and provenance repositories as previous node that data will get loaded back in on startup and processing continue where it left off.  Remember that each nodes repositories are unique to that node (meaning you can't combine them and they don't all contain the same content). Another thing to review is the release notes.  https://cwiki.apache.org/confluence/display/NIFI/Release+Notes You'll want to review all the release notes between 1.12 and 1.17.  Apache NiFi is known to deprecate and remove some components (processors, controller services, reporting tasks, etc) from time to time.  You'll want to check to see if any components you use in your current dataflows are being removed. Additional some components may have changed typically resulting in additional properties being added.  When you start with the newer version of NiFi, it will load your existing flow.xml.gz (1.17 will actually generate a flow.json.gz file from your flow.xml.gz) and upgrade all your components to use the newer 1.17 version of the component classes.  So you'll want to review you flow after upgrade to make sure none of your components that were previous valid have become invalid because new property exist that must be configured. NOTE: 1.17 will use start using the flow.json.gz once upgrade, as the flow.xml.gz format is deprecated. If you found this response assisted you with your query, please take a moment to login and click on "Accept as Solution" below this response. Thank you, Matt   ... View more