@leandrolinof  The "EvaluateJsonPath" processor you shared has the configured destination as a FlowFile-Attribute and thus leaves the content of the FlowFile unchanged. So if FlowFile Attribute is where you want this parsed output to reside, you could use "ExtractText" [1] as an alternative solution. The used value here would be: .*Erros":\[(.*)\].* This has only 1 capture group which is for the content you are trying to extract.  That content is then added to the FlowFile in a new FlowFile attribute based on the property name.   If you instead want to replace the content of your FlowFile with only the portion of the original content you are trying to extract, you could use the "ReplaceText" [2] processor. The used Java regex "Search value" here would be: (.*Erros":\[)(.*)(\].*$) Which breaks source content into 3 capture groups so regex matches entire content and capture group 2 matches on the string output you are looking for. So "Replacement Value" is simply set to "$2" so that entire content is replaced with just contents of capture group 2. [1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.ExtractText/index.html [2] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.ReplaceText/index.html   If you found this help you with your question, please take a moment to login and click accept on this solution. Thanks, Matt ... View more

@Justee  You definitely can't use "/Home/Data/" in the "Remote File" property of the FetchSFTP processor.  This must be a the full path to a specific file being fetched, which means it should be getting that path and filename values from the inbound FlowFile's attributes. Your ListSFTP (which you said is working) would be producing a FlowFile for each File found at target directory non your SFTP server.  Those FlowFiles will get queued on the outbound connection containing the ListSFTP's "success" relationship. That connection is the inbound connection to the FetchSFTP processor that will connect to same SFTP server and retrieve the content for each of those FlowFiles that were produced by the ListSFTP processor. The "Remote File" property on the FetchSFTP processor should be using NiFi Expression Language (EL) to dynamically set the path and filename being fetched uniquely for each inbound FlowFile. ${path}/${filename} NiFi is case sensitive, so make sure you are using all lowercase so it matches the attribute names created on the Source FlowFiles by the ListSFTP processor. When you say the FetchSFTP processor is "not working", what does that mean? To which relationship are the inbound FlowFiles getting routed when it does not work? What exception (ERROR) is logged in the nifi-app.log when it fails to fetch the content for an inbound FlowFile? Make sure when you configured the username and password on the FetchSFTP you did not accidentally add a leading or trailing whitespace. Hope this helps, Matt ... View more

@robnew    The "CompressContent" [1] processor can be used to decompress gz files. My suggestion here since only some log files are compressed is to set up a flow that passes these FlowFiles through an "IdentifyMimeType" [2] processor.  This will write out a new mime.type attribute on the FlowFiles.  Then use a "RouteOnAttribute" [3] processor to route FlowFiles with mime.type = application/gzip (each new dynamic property becomes a new outbound relationship) to that "CompressContent" processor and the "unmatched" relationship (which will have every other non gz file) on through your flow without passing through "CompressContent" processor.    [1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.CompressContent/index.html [2] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.IdentifyMimeType/index.html [3] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.RouteOnAttribute/index.html If you found this helpful with your question, please take a moment to login and click accept on this solution. Thanks, Matt ... View more

@Dmitry  Trying to fully understand your use case here... You have 1 to many FlowFile where you want to merge the content of each of those FlowFiles in to a single FlowFile if all those FlowFile have an attribute " manager_name" with the same value? If so, the MergeContent [1] or MergeRecord [2] processors could do that for you.  You would set the "Correlation Attribute Name" property in those processors the place incoming FlowFiles in to bins based on unique values set for "manager_name" on each incoming FlowFile.  Those bins are then merged based on criteria set in other properties on the processor. The merged output FlowFile will have several new attribute written to it.  One of those new attributes is "merge.count" which will contain the count of the number fo input FlowFiles that were merged.   You could use an UpdateAttribute processor to read the value from "merge.count" and "manager_name", then assign those to other attribute named "total" and "general key" is those attribute names are specifically needed. [1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.MergeContent/index.html [2] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.MergeRecord/index.html   If you found this information helped with you question, please take a moment to login and click accept on this solution. Matt ... View more

@Justee  So with NiFi, there is a UI, but you need to make sure there is a route to the host (where you are laughing your browser) and target: <hostname/IP>:<port>  If your NiFi in startup is binding to private network interface or only the loopback IP address, you would most likely be unable to reach those IPs from your client machine on a public network. From your client machine, can you ping that "<hostname/IP>"? Does the server where your NiFi is running have multiple network interfaces for perhaps a public and a private network?  Did you try changing the IP that NiFi binds to in nifi.properties file: nifi.web.http.host= Did you try to tell your NiFi to bind to multiple network interfaces in the nifi.properties file if they exist in that server?  For example: nifi.web.http.network.interface.eth0=eth0 nifi.web.http.network.interface.eth1=eth1 Your servers network interfaces may be different then above example. If you found this solution helpful, please take a moment to login in and click accept on all solutions that helped you. Thanks, Matt ... View more

@pherschmann    The " nifi is running but not responding to ping requests" is not referring to ability to ping the host. When NiFi is started, you are starting the NiFi bootstrap process. This bootstrap process then starts a child NiFi process which is what runs the main NiFi that you interact with.  When you run the status command, it us asking the bootstrap process to return the status of that child process.  It is telling you that it sees that the PID for that child process still exists so reports it is running, but when it tries to get a response from that child process it fails and logs that the status ping got no response. This almost always is the result of some resource contention within NiFi.  The NiFi process may have been going through Java Garbage Collection (GC) at the time you ran the status command. (all GC is stop-the-world event, so JVM will not respond to anything else), or thread usage (CPU consumption) was high at the time, etc... If you see this allot or are seeing node disconnection or UI stability issues, you may want to start monitoring resources on the host, or maybe enable GC logging in your NiFi to see if the service is spending a lot of time doing GC.  Increasing heap allocated to NiFi or making dataflow design changes to reduce the heap memory footprint of your current dataflow(s) design would help there. Hope you found this information helpful. If so, please take a moment to login and click accept on this solution, Matt ... View more

@hkh    The appender you shared is not valid. You have configured your appender rolling policy to use: SizeAndTimeBasedRollingPolicy However, your file naming patten only supports time based pattern ${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d.log This leaves you will two options: Option1: - Keep using the "SizeAndTimeBasedRollingPolicy", but change your file naming pattern - A pattern like "${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd}.%i.log" The "{yyyy-MM-dd}" is option but allows you to specify the date format. - With above pattern logback will retain 1 day of log history, but may or may not have more then 1 log in a given day depending on volume of logging that is occurring.  If a daily log reaches the configured "maxFileSize" the log with roll.  This allows you to keep you logs at manageable sizes.  When the log rolls it will get a one up number applied per this new file naming pattern. For example: nifi.app_2021-04-28.1.log nifi.app_2021-04-28.2.log - While this can still result in an unbounded number fo incremental logs files created in a single day, you can control overall disk usage by adding another property within the "rollingPolicy" section that will start purging incremental rolled logs if max amount of space consumed by these rolled logs exceeds this set value.  Add this line below your " < maxHistory >1 </ maxHistory >" line: <totalSizeCap>3GB</totalSizeCap> Note: Will only remove rolled/archive logs and will not remove active log. Option 2: - Change the rolling policy you are using to "ch.qos.logback.core.rolling.TimeBasedRollingPolicy" - With this policy you can keep the file name pattern you already have "${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d.log" - You will need to comment out remove the following line: <maxFileSize>1GB</maxFileSize> it is only valid for size based rolling policies. - The downside of this setup is that a single daily log file can grow unbounded. As far as your other question goes.... The logback.xml is the only configuration file in NiFi that you can edit and its changes will take affect without needing a NiFi restart.  Some caveats... NiFi did not write logback and it certainly has its quirks.  For example... If you enter a bad configuration, it may simply just stop logging.   The way maxHistory works in logback prevents the cleanup from looking at older rolled logs outside the maxHistory window, so you will need to cleanup the older rolled logs manually initially and not expect logback to do that clean-up for you after editing the logback.xml.  Since you are making big changes to file naming pattern and potentially the rolling policy, i'd encourage you to restart NiFi anyway so it cleanly starts writing to the new log format on startup. Hope you found this helpful. If so take a moment to login and click accept on this solution, Matt ... View more

@Vickey  The file filter property of the unpackContent processor takes a java regular expression and can be used when unpacking tar or zip file. In your unpackContent processor, set the "Packaging format" to either "ZIP" or "TAR" based on what package format is used by your source file. The set a java regular expression such as below to extract only files within that package where the filename ends with the .csv, .txt, or .xml extensions: .*\.(txt|xml|csv) Hope this helps, Matt   ... View more

@ram_g  With all 100 FlowFiles committed to the success relationship of your custom processor at the same time, how do we want NiFi to determine their priority order? If you can out put some attributes on each FlowFile that your custom processor is creating, those attribute values could be used set processing order downstream. Hope this helps, Matt ... View more

@john  I have a HDF 3.4.1.1 cluster (Based off NiFi 1.11.4) setup and with PGs version controlled and can change processors from started to stopped to disabled without it triggering a local change.  However, HDF 3.4.1.1 ships with NiFi-Registry 0.3 and not 0.8. I have another HDF 3.5.2 cluster (based off NiFi 1.12.1) and ships with NiFi-Registry 0.8.  In that cluster, I can also change a processor from start to stop to disabled and it does trigger a local change. I see someone filled a Jira about this change in behavior: https://issues.apache.org/jira/browse/NIFI-8160 The tracking of Enabled and Disabled State in NiFi-Registry was added as part of: https://issues.apache.org/jira/browse/NIFI-6025 Hope this helps, Matt ... View more

@AnkushKoul  Since the 30 seconds since last execution has past, the processor is available to be immediately scheduled once a thread becomes available.  So second thread would not wait till 60 seconds.  This setting is minimum wait between executions. Other factors come int play that can affect component execution scheduling. NiFi hands out threads to processors from the Max Timer Driven Thread Count resource pool set via Controller Settings under the global menu in the upper right corner.  Naturally you will have more components on your canvas then the size of this resource pool (which should be set initially to only 2-4 times the number fo cores you have on a single node since setting applies per node).  NiFi will hand these available threads out to processors requesting CPU time to execute.  Most component threads are in the range of milliseconds of execution, bit some can be more resource intensive and take longer to complete.  Before increasing this resource pool, you should monitor the CPU impact/usage with all your dataflows running. Then make small increments if resource exist. Hope this answers your questions.  If, so please take. moment to accept the answer(s) that helped. Matt ... View more