@Vickey  The file filter property of the unpackContent processor takes a java regular expression and can be used when unpacking tar or zip file. In your unpackContent processor, set the "Packaging format" to either "ZIP" or "TAR" based on what package format is used by your source file. The set a java regular expression such as below to extract only files within that package where the filename ends with the .csv, .txt, or .xml extensions: .*\.(txt|xml|csv) Hope this helps, Matt   ... View more

@ram_g  With all 100 FlowFiles committed to the success relationship of your custom processor at the same time, how do we want NiFi to determine their priority order? If you can out put some attributes on each FlowFile that your custom processor is creating, those attribute values could be used set processing order downstream. Hope this helps, Matt ... View more

@john  I have a HDF 3.4.1.1 cluster (Based off NiFi 1.11.4) setup and with PGs version controlled and can change processors from started to stopped to disabled without it triggering a local change.  However, HDF 3.4.1.1 ships with NiFi-Registry 0.3 and not 0.8. I have another HDF 3.5.2 cluster (based off NiFi 1.12.1) and ships with NiFi-Registry 0.8.  In that cluster, I can also change a processor from start to stop to disabled and it does trigger a local change. I see someone filled a Jira about this change in behavior: https://issues.apache.org/jira/browse/NIFI-8160 The tracking of Enabled and Disabled State in NiFi-Registry was added as part of: https://issues.apache.org/jira/browse/NIFI-6025 Hope this helps, Matt ... View more

@AnkushKoul  Since the 30 seconds since last execution has past, the processor is available to be immediately scheduled once a thread becomes available.  So second thread would not wait till 60 seconds.  This setting is minimum wait between executions. Other factors come int play that can affect component execution scheduling. NiFi hands out threads to processors from the Max Timer Driven Thread Count resource pool set via Controller Settings under the global menu in the upper right corner.  Naturally you will have more components on your canvas then the size of this resource pool (which should be set initially to only 2-4 times the number fo cores you have on a single node since setting applies per node).  NiFi will hand these available threads out to processors requesting CPU time to execute.  Most component threads are in the range of milliseconds of execution, bit some can be more resource intensive and take longer to complete.  Before increasing this resource pool, you should monitor the CPU impact/usage with all your dataflows running. Then make small increments if resource exist. Hope this answers your questions.  If, so please take. moment to accept the answer(s) that helped. Matt ... View more

@Garyy  You are correct.  Since NiFi does not use sessions as mentioned in my last response, the client must authenticate every action performed.  When you "login" to NiFi, the result is a bearer token being issued to the user which your browser stores and reuses in all subsequent request to the NiFi endpoints.  At the same time a server side token for your user is also stored on the specific NIFi node you logged in to.  The configuration in your NiFi login provider dictates how long those bearer tokens are good for.  With your setting of 1 hour, you would be forced to re-login again every hour. Thanks, Matt ... View more

@Garyy    So the first place you may want to start is opening developer tools in your browser and then trying to connect your NiFi UI and take note of what calls are taking the longest to return and which it eventually times out on. You may also want to enable garbage collection logging within your NiFi JVM (do this by adding GC logging java args in the NiFI bootstrap.conf file).   If your JVM is encountering long and/or frequent GC (all GC events are stop-the-world events), this can result in timeouts made to the UI. It is also not entirely clear to me what you mean by your UI access get terminated.  NiFi does not use sessions. What are you observing? Can you provide more detail and screenshots?   Matt ... View more

@sambeth  The hash (#) character is reserved as a delimiter to separate the URI of an object from a fragment identifier. Registry has a number of different fragment Identifiers.  The fragment identifier represents a part of, fragment of, or a sub-function within, an object.  The fragment identifier follows the "/#/" in the URL and can  represent fragments in text documents by line and character range, or in graphics by coordinates, or in structured documents using ladders.  For example the "grid-list" of flows displayed when you access the NiFi UI. No, you cannot remove the # from the URL.  Are you encountering an issue? Hope this helps, Matt ... View more

@Pavitran  This does present a challenge. Typically the ListFile is used to list files from a local file system.  That processor component is designed record state (default based on last modified timestamp) so that only newer files are consumed.   But the first run would result in listing all files by default.  Also looking at your example, your latest directory does not correspond to current day.  The listFile (does not actually consume to content) generates a 0 byte FlowFile for each file listed along with some attributes/metadata about the source file.  The FetchFile processor would then be used to fetch the actual content, this allows with large listing to redistribute these 0 byte FlowFiles across all nodes in your cluster before consuming the content (provided same local file system is mounted across all nodes.  If different files per node, do not load balance between processors). So you could make a first run which lists everything and just delete those 0 byte files.  That would establish state.  Then from that point on the ListFile would only list the newest files created.   Pros: 1. State allows this processor to be unaffected by outages, the processor will still consume latest all non previously listed files after an outage. Cons: 1. You have this initial run which would create potentially a lot of 0 byte FlowFiles to get rid of in order to establish state. 2. With an extended outage, on restart of the flow it may consume more than just the latest since it will be consuming all files with newer timestamps than timestamp last stored in state. Other options: A: The ListFile processor has an optional property that sets the " Maximum File Age" which limits the listing of Files to those not olde then x set amount of time. Pros to. setting this property: 1. Reduces or eliminates the massive listing on first run  Cons to setting this property: 2. Under an extended outage, where outage exceeds configured " Maximum File Age", a file you wanted listed may be skipped. B: Since the FetchFile uses attributes/metadata from to fetch the actual content, you could craft a source FlowFile on your own and send it to the FetchFile processor.   For example, use a ExecuteStreamCommand processor execute a bash file on disk to get the list of Files only from the latest directory.  Then use UpdateAttribute to add the other required attributes needed by FetchFile to get the actual content.  Then use SplitFile to split that listing of files in to individual FlowFiles before the FetchFile processor. Pros:  1. You are in control of what is being listed. Cons: 1. Depending on how often a new directory is created and how often you run your ExecuteStreamCommand processor, you may end up listing the same source files over again since you will not have a state option with ExecuteStreamCommand.  But you may be able to handle this via detectDuplicate processor in your flow design. 2. If the listed Directory has some new file added to it post previous listing by ExecuteStreamCommand, next run will list all previous files again along with new ones for same directory.  Again, might be able to handle this with detectDuplicate processor. Hope this helps give you some ideas, Matt ... View more