@Thangarajan Pannerselvam If your GetFTP processor is configured with "delete original" set to false, every time this processor runs it will pull all th the files it finds including those pulled in the last run of the getFP processor. The ListFTP processor maintains state unlike the GetFTP processor. so if you replace your GetFTP with both ListFTP and FetchFTP processors, you will not see the same files pulled twice unless the timestamp on the files on the FTP server are updated. Thanks, Matt ... View more

@mayki wogno Same question as this thread: https://community.hortonworks.com/questions/88962/nifi-processor-not-the-most-up-to-date.html ... View more

@nyakkanti FlowFiles consist of FlowFile attributes and FlowFile content. - FlowFile attributes are kept in heap during processing and persisted to the FlowFile repository. - FlowFile content is kept in claims within the content repository. A claim is moved is moved to archive once their no longer exists any FlowFiles still active anywhere in your dataflow pointing at it. Archiving is enabled by default but can be disabled in the nifi.properties file: nifi.content.repository.archive.enabled=true If you disable archiving, the claim is purged from NiFi's content repository rather the being archived. What is important to understand is how claims work. By default in the nifi.properties file, claims can contain up to 100 FlowFiles or a min 10 MB of data (whichever occurs first). So a claim will not be purged until every piece of content in that claim has completed processing. As long as just one piece of content in that claim is still referenced, the entire claim will still exist in the content repository. As far as FlowFile attributes are concerned, they are persisted in NiFi provenance based on the configured retention in the nifi.properties file. You can perform provenance searches within NiFi to return FlowFile history and look at the attributes of those FlowFiles at any point int their lineage. Thanks, Matt ... View more

@Mohammed El Moumni Each Node in a NiFi cluster runs its own copy of the dataflow and works on its own set of FlowFiles. Looking at the screenshot you have above of your queue list, you can see that the two FlowFiles are not on the same node. So each node is running a MergeContent processor and each node is waiting for another FlowFile to complete their bins. You will need to look back earlier in your dataflow to see how your data is being ingested by your nodes to make sure that the matching sets of files end up on the same node for merging. Thanks, Matt ... View more

@matthew N Any time you see the following: Unable to perform the desired action due to insufficient permissions. Contact the system administrator. You are having an authorization issue and not an authentication issue. If you look tail your nifi-app.log while you try to login again, you will see two lines output. One will state successful authentication. 2017-03-14 14:36:43,402 INFO [NiFi Web Server-5418] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<cn=Matt,ou=People,dc=sme,dc=nifi><CN=nifi-11.openstacklocal, OU=SME, O=NIFI, L=Baltimore, ST=MD, C=US><CN=nifi-13.openstacklocal, OU=SME, O=HWX, L=Baltimore, ST=MD, C=US>) GET https://nifi-11.openstacklocal:9091/nifi-api/flow/config (source ip: x.x.x.x) 2017-03-14 14:36:43,402 INFO [NiFi Web Server-5418] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for cn=Matt,ou=People,dc=sme,dc=nifi Then you will see a line that contains the above response. You need to make sure that the DN shown in this log matches exactly with what you had configured for your "Initial Admin Identity". You entry in LDAP may have uppercase CN, OU, ..., but in your nifi use log you may see lower case cn=, ou, ... It must match what is in the user.log. You can also take a look in the users.xml file that NiFi generates to make sure your user exists. It will have a UUID assigned to your user. Take that UUID and check which resources that UUID has been granted access to in the authorizations.xml file. For an ADMIN user, you will need at a minimum: /flow /controller /policies /tenants Thanks, Matt ... View more

@Christophe Vico Where in the UI did you look for the controller service? Controller services that are used by processors such as HIveConnectionPool are created within process groups and not at the core level found in the hamburger menu. You can access controller services within process groups by selecting the process group and then clicking on the "gear" icon in the operate panel: Thanks, Matt ... View more