@Justee  You definitely can't use "/Home/Data/" in the "Remote File" property of the FetchSFTP processor.  This must be a the full path to a specific file being fetched, which means it should be getting that path and filename values from the inbound FlowFile's attributes. Your ListSFTP (which you said is working) would be producing a FlowFile for each File found at target directory non your SFTP server.  Those FlowFiles will get queued on the outbound connection containing the ListSFTP's "success" relationship. That connection is the inbound connection to the FetchSFTP processor that will connect to same SFTP server and retrieve the content for each of those FlowFiles that were produced by the ListSFTP processor. The "Remote File" property on the FetchSFTP processor should be using NiFi Expression Language (EL) to dynamically set the path and filename being fetched uniquely for each inbound FlowFile. ${path}/${filename} NiFi is case sensitive, so make sure you are using all lowercase so it matches the attribute names created on the Source FlowFiles by the ListSFTP processor. When you say the FetchSFTP processor is "not working", what does that mean? To which relationship are the inbound FlowFiles getting routed when it does not work? What exception (ERROR) is logged in the nifi-app.log when it fails to fetch the content for an inbound FlowFile? Make sure when you configured the username and password on the FetchSFTP you did not accidentally add a leading or trailing whitespace. Hope this helps, Matt ... View more

@syntax_  NiFi does not provide a method for uploading templates in bulk.   But anything you can do via the UI, you can also do via rest-api calls [1] through curl.  So you could script the bulk upload through rest-api external to NiFi's UI. I would strongly discourage bulk uploading templates to NiFi.  Templates should be uploaded as needed, instantiate to the canvas and then template deleted from NiFi.   All uploaded templates even if not instantiated to the canvas become part of the flow.xml loaded in to heap memory.  So keeping a large number of templates uploaded to your NiFi can have a considerable impact on JVM heap memory usage because now you have not only your active flow on the canvas in memory, but also these templates. Additionally, In a NiFi cluster on startup the flow.xml.gz is uncompressed, loaded in to heap memory, and a flow fingerprint created on every node.  These flow fingerprints are then compared to make sure all nodes joining the cluster are running with the same flow.xml.  Since these templates can just add unnecessary size to the flow.xml.gz, this can impact startup times and flow fingerprint comparison time. A better approach is to get your reusable flow migrated into NiFi-Registry [2].  You can have 1 to many NiFi's all connect to a single NiFi-Registry where all your flows exist.  Then all you need to do is drag a process group to the canvas and select "import" which will allow you to select one of these flows from NiFi-Registry and add it to your canvas.  Now you have your flow without the additional impact on heap from having it in NiFi both as a template and as a dataflow on your canvas with easy access to load same flow over and over.  I encourage you to explore NiFi-Registry. [1] http://nifi.apache.org/docs/nifi-docs/rest-api/index.html [2] https://nifi.apache.org/registry.html   If you found this response helped with your query, please take a moment to login in and click accept on this solution. Thanks, Matt ... View more

@Justee  So with NiFi, there is a UI, but you need to make sure there is a route to the host (where you are laughing your browser) and target: <hostname/IP>:<port>  If your NiFi in startup is binding to private network interface or only the loopback IP address, you would most likely be unable to reach those IPs from your client machine on a public network. From your client machine, can you ping that "<hostname/IP>"? Does the server where your NiFi is running have multiple network interfaces for perhaps a public and a private network?  Did you try changing the IP that NiFi binds to in nifi.properties file: nifi.web.http.host= Did you try to tell your NiFi to bind to multiple network interfaces in the nifi.properties file if they exist in that server?  For example: nifi.web.http.network.interface.eth0=eth0 nifi.web.http.network.interface.eth1=eth1 Your servers network interfaces may be different then above example. If you found this solution helpful, please take a moment to login in and click accept on all solutions that helped you. Thanks, Matt ... View more

@Arash  A FlowFile consists of to parts: - FlowFile content - Content resides on disk in the content repository and not in heap memory.  Some components used may need to load content in to memory to perform the function of that component. - FlowFile Attributes/metadata - FlowFiles actively queued in a connection will have their attributes/metadata held in heap memory.  Swapping is the only mechanism that can move this FlowFile metadata/attribute data out of heap to swap on disk. It is important to remember that MiNiFi will only start swapping FlowFiles to disk once the swap threshold  per connection reaches the configured value(default 20,000).  Swap files are created in batches of 10,000.  So in a smoothly running flow there should be very little, if any, swapping of FlowFile attributes/metadata happening.  This should only be happening at times of data bursts. To keep heap usage down, limit the size of your connection queue backpressure object threshold.  The default is 10000 which means a connection would never accumulate enough FlowFiles to trigger a swap file anyway normally (backpressure is a soft limit, so if a source processor is allowed to execute because the downstream connection is not applying backpressure yet and that source processor execution results in 30,000 FlowFiles being created, then all 30,000 are placed on downstream connection which would result in swap files being created). When you are building your dataflow via NiFi that you will use on your MiNiFi agent, be mindful of above and and look at the embedded documentation for the components you will be using in that dataflow.  The embedded docs include resource consideration section under each component if there are known impacts on heap memory or cpu.  Processors that merge or split FlowFiles commonly used can have an impact on heap memory if not configured wisely. Hope this helps remove some concern and provide useful insight. If you found this helpful, please take a moment to login and click accept on this solution. Matt ... View more