@spartakus  You will want to review the release notes for all release between 1.23. and 1.28.1  https://cwiki.apache.org/confluence/display/NIFI/Release+Notes/#ReleaseNotes-Version1.23.0 You'll notice a number of components deprecated. Deprecated components are not removed, but that are marked as deprecated and will log if used that they are deprecated.  Deprecated components will eventually be removed form Apache NiFi (most removed in NiFi 2.0). You'll also make note of any items in the release notes that state:   Moved <XYZ> to optional build profiles   This means that these components were removed from Apache NiFi.   These components can be added manually later by downloading the nars from the central maven repository and adding them to the NiFi lib directory if you need them still.  https://mvnrepository.com/artifact/org.apache.nifi Make a back up of your flow.json.gz, nifi.properties, and optionally flow.xml.gz files before you upgrade. Otherwise, I see no issues with upgrading from 1.22 -- > 1.28 directly. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@itninja, Did the response help resolve your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future. However, if you still have concerns, could you please provide the information that @MattWho has requested? ... View more

@MattWho  Thank you for explanation. Now I understand MergeRecord determines to which file each flowfile to be merged by schema information. I'll consider increasing "Minimum number of records" as you recommended. Thanks, ... View more

It seems that the next scheduled task has not been triggered         ... View more

@Eslam  Welcome to the community. In order to get helpful answers, you'll need to provide more detail around your use case. NiFi provides many processors for connecting to various services on external hosts. You can find the list of default processors available with the Apache NiFi release here: NiFi 1.x release: https://nifi.apache.org/docs/nifi-docs/ NiFi 2.x release: https://nifi.apache.org/components/ At the very basic of level you have processors like: GetSFTP ListSFTP / FetchSFTP But there are also processor for connecting SMB, Splunk, rest-apis, SNMP, FTP, DBs, Kafka, hive, etc. on external servers. Look through the components list in the documentation for Get, List, Fetch, and Query type processors to see if any of them meet your use case needs. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

Thanks @MattWho . Checking with mergeContent option with our data volume, if it works then will go with this. Thanks for help/suggestion.   ... View more

I changed my config to this: This seems to do the trick. ... View more

@ShellyIsGolden  500k+ files is a lot to list and the lookup on subsequent runs to look for new files.  A few questions first: How is your listSFTP processor scheduling configured? With the Initial listing, how long does it take to output he 500K+ FlowFiles from time processor is started? When files are added to the SFTP server, are they added using a dot rename method? Is the last modified timestamp being updated on the files as they are being written to the SFTP server? So the processor when executed for the initial time will list all files regardless of the configured "Entity Tracking Time Window" set value.  Subsequent executions will only list files with and last modified timestamp within the configured "Entity Tracking Time Window" set value.  So accurate last modified timestamps are important. With initial listing of a new processor (or copy of existing processor) there is no step to check list files against the cache entries to see if file has never been listed before or if a listed file has changed in size since last listed.  This lookup and comparison does happen on subsequent runs and can use considerable heap.  Do you see any OutOf Memory (OOM) exceptions in your NiFi app logs? Depending on how often the processor executes, consider reducing the configured "Entity Tracking Time Window" value so fewer files are listed in the subsequent executions that need to be looked up.  Set it to what is needed with a small buffer between each processor execution.  Considering that it sounds you have yoru processor scheduled to execute every 1 minute, maybe try setting this to 30 minutes instead to see what impact it has. When you see the issue, does the processor show an active thread in the upper right corner that never seems to go away? When the issue appears, rather then copy the processor, what happens if you simply stop the processor (make sure all active threads complete, shows no active threads number in upper right corner of processor) and then just restart it? In the latest version of Apache NiFi, a "Remote Poll Batch Size" property (defaults to 5000) was added to the listSFTP processor which may help here considering the tremendous amount files being listed in your case. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more