@Allen123    Based on your description, the ListSftp processor is working exactly as designed and configured. You have told it to list file from both directories and have also told it to search for both filenames in those directories. There is no way to configure the processor to ignore the extra unwanted a.text or b.text files. You have two options: Option 1: - Create to listSFTP processor with each configured to list within a unique directory and "Search Recursively" set to false.  Then set a "File Filter Regex" so it only lists the desired file from each directory. - The feed the success relationship from each of those two ListSftp processors to a single FetchSftp processor. Option2: The ListSftp processor is designed to only list files and not actually consume any content.  The content of the files listed is consumed later using the FetchSftp processor.  Between the ListSftp and the FetchSftp processors you could insert a RouteOnAttribute [1] processor that you could configure to only pass the two specific files you want on to FetchSftp while auto-terminating the others.   You would route the matched relationship via a connection to the FetchSFTP and "auto-terminate" the unmatched relationship. [1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.RouteOnAttribute/index.html If you find that this has helped with your query, please take a moment to login and click accept on the solution. Thanks, Matt ... View more

@leandrolinof  The "EvaluateJsonPath" processor you shared has the configured destination as a FlowFile-Attribute and thus leaves the content of the FlowFile unchanged. So if FlowFile Attribute is where you want this parsed output to reside, you could use "ExtractText" [1] as an alternative solution. The used value here would be: .*Erros":\[(.*)\].* This has only 1 capture group which is for the content you are trying to extract.  That content is then added to the FlowFile in a new FlowFile attribute based on the property name.   If you instead want to replace the content of your FlowFile with only the portion of the original content you are trying to extract, you could use the "ReplaceText" [2] processor. The used Java regex "Search value" here would be: (.*Erros":\[)(.*)(\].*$) Which breaks source content into 3 capture groups so regex matches entire content and capture group 2 matches on the string output you are looking for. So "Replacement Value" is simply set to "$2" so that entire content is replaced with just contents of capture group 2. [1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.ExtractText/index.html [2] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.ReplaceText/index.html   If you found this help you with your question, please take a moment to login and click accept on this solution. Thanks, Matt ... View more

@Justee  You definitely can't use "/Home/Data/" in the "Remote File" property of the FetchSFTP processor.  This must be a the full path to a specific file being fetched, which means it should be getting that path and filename values from the inbound FlowFile's attributes. Your ListSFTP (which you said is working) would be producing a FlowFile for each File found at target directory non your SFTP server.  Those FlowFiles will get queued on the outbound connection containing the ListSFTP's "success" relationship. That connection is the inbound connection to the FetchSFTP processor that will connect to same SFTP server and retrieve the content for each of those FlowFiles that were produced by the ListSFTP processor. The "Remote File" property on the FetchSFTP processor should be using NiFi Expression Language (EL) to dynamically set the path and filename being fetched uniquely for each inbound FlowFile. ${path}/${filename} NiFi is case sensitive, so make sure you are using all lowercase so it matches the attribute names created on the Source FlowFiles by the ListSFTP processor. When you say the FetchSFTP processor is "not working", what does that mean? To which relationship are the inbound FlowFiles getting routed when it does not work? What exception (ERROR) is logged in the nifi-app.log when it fails to fetch the content for an inbound FlowFile? Make sure when you configured the username and password on the FetchSFTP you did not accidentally add a leading or trailing whitespace. Hope this helps, Matt ... View more

@syntax_  NiFi does not provide a method for uploading templates in bulk.   But anything you can do via the UI, you can also do via rest-api calls [1] through curl.  So you could script the bulk upload through rest-api external to NiFi's UI. I would strongly discourage bulk uploading templates to NiFi.  Templates should be uploaded as needed, instantiate to the canvas and then template deleted from NiFi.   All uploaded templates even if not instantiated to the canvas become part of the flow.xml loaded in to heap memory.  So keeping a large number of templates uploaded to your NiFi can have a considerable impact on JVM heap memory usage because now you have not only your active flow on the canvas in memory, but also these templates. Additionally, In a NiFi cluster on startup the flow.xml.gz is uncompressed, loaded in to heap memory, and a flow fingerprint created on every node.  These flow fingerprints are then compared to make sure all nodes joining the cluster are running with the same flow.xml.  Since these templates can just add unnecessary size to the flow.xml.gz, this can impact startup times and flow fingerprint comparison time. A better approach is to get your reusable flow migrated into NiFi-Registry [2].  You can have 1 to many NiFi's all connect to a single NiFi-Registry where all your flows exist.  Then all you need to do is drag a process group to the canvas and select "import" which will allow you to select one of these flows from NiFi-Registry and add it to your canvas.  Now you have your flow without the additional impact on heap from having it in NiFi both as a template and as a dataflow on your canvas with easy access to load same flow over and over.  I encourage you to explore NiFi-Registry. [1] http://nifi.apache.org/docs/nifi-docs/rest-api/index.html [2] https://nifi.apache.org/registry.html   If you found this response helped with your query, please take a moment to login in and click accept on this solution. Thanks, Matt ... View more

@robnew    The "CompressContent" [1] processor can be used to decompress gz files. My suggestion here since only some log files are compressed is to set up a flow that passes these FlowFiles through an "IdentifyMimeType" [2] processor.  This will write out a new mime.type attribute on the FlowFiles.  Then use a "RouteOnAttribute" [3] processor to route FlowFiles with mime.type = application/gzip (each new dynamic property becomes a new outbound relationship) to that "CompressContent" processor and the "unmatched" relationship (which will have every other non gz file) on through your flow without passing through "CompressContent" processor.    [1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.CompressContent/index.html [2] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.IdentifyMimeType/index.html [3] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.RouteOnAttribute/index.html If you found this helpful with your question, please take a moment to login and click accept on this solution. Thanks, Matt ... View more