@Joshua Adeleke With NiFi dataflows it is typically to see large numbers of open file handles and user processes because of the concurrent thread operation supported by its many components. In many cases you will find that the default ulimits for both open files and processes will fall short of what is needed for most dataflows. I recommend setting these values to 50000 out the gate. You may find your self depending on volumes and complexity of your dataflow(s) needing to even set this higher. The default 1024 is almost always guaranteed to be an issue. /etc/security/limits.conf * hard nproc 50000 * soft nproc 50000 * hard nofiles 50000 * soft nofiles 50000 /etc/security/limits.d/90-nproc.conf * hard nproc 50000 * soft nproc 50000 Thanks, Matt ... View more

@mayki wogno What type of downstream processor is the queue you are trying to empty connect to? Some processors such as the MergeContent processor when running have ownership of FlowFiles in the incoming queue. The MergeContent assigns FlowFiles on its incoming queue(s) to bins. You will not be able to clear the queue of any FlowFiles that are currently assigned to a bin. If you stop the processor downstream of your queue (Processor must show no running threads), can you then successfully empty the queue? Thanks, Matt ... View more

@mayki wogno How are you executing your provenance query? Are you selecting "Data Provenance" from within the hamburger menu in the upper right of the Ui or are you selecting "Data provenance" from the context menu that appears by right clicking on your listHDFS processor? The above performs a global provenance query of all your dataflows by default unless you add a filter. triggering a provenance query through a specific components context menu will add a filter based upon that components UUID. Thanks, Matt ... View more

@Thangarajan Pannerselvam If your GetFTP processor is configured with "delete original" set to false, every time this processor runs it will pull all th the files it finds including those pulled in the last run of the getFP processor. The ListFTP processor maintains state unlike the GetFTP processor. so if you replace your GetFTP with both ListFTP and FetchFTP processors, you will not see the same files pulled twice unless the timestamp on the files on the FTP server are updated. Thanks, Matt ... View more

@mayki wogno Same question as this thread: https://community.hortonworks.com/questions/88962/nifi-processor-not-the-most-up-to-date.html ... View more

@nyakkanti FlowFiles consist of FlowFile attributes and FlowFile content. - FlowFile attributes are kept in heap during processing and persisted to the FlowFile repository. - FlowFile content is kept in claims within the content repository. A claim is moved is moved to archive once their no longer exists any FlowFiles still active anywhere in your dataflow pointing at it. Archiving is enabled by default but can be disabled in the nifi.properties file: nifi.content.repository.archive.enabled=true If you disable archiving, the claim is purged from NiFi's content repository rather the being archived. What is important to understand is how claims work. By default in the nifi.properties file, claims can contain up to 100 FlowFiles or a min 10 MB of data (whichever occurs first). So a claim will not be purged until every piece of content in that claim has completed processing. As long as just one piece of content in that claim is still referenced, the entire claim will still exist in the content repository. As far as FlowFile attributes are concerned, they are persisted in NiFi provenance based on the configured retention in the nifi.properties file. You can perform provenance searches within NiFi to return FlowFile history and look at the attributes of those FlowFiles at any point int their lineage. Thanks, Matt ... View more