@Anishkumar Valsalam Great. Yes you can use the same approach. To have load balancing you need lot of files. If there's only few files, NiFi will not balance because it optimizes with batching. Read this article to understand how it works : https://community.hortonworks.com/articles/109629/how-to-achieve-better-load-balancing-using-nifis-s.html Thanks ... View more

@Anishkumar Valsalam Use this in the Remote File attribute of FetchSFTP ${path}/${filename} To test you need to right click on ListSFTP and click on forget state ... View more

The error is that you are trying to access to a file *.txt which doesn't exist. You should use the filename property instead of *.txt. This property will be set by the list processor as an attribute ... View more

Did you get any errors? can you add a file to your FTP server and see if NiFi get it? what's the scheduling frequency you have at List processor ? ... View more

Hi @balalaika Yes you can use a Cron to do it twice per day. The following example will run at 3:00am and 3:05am each day. 0 5,0 3 * * ? Regarding the logic of the processor in previous versions, I understand that the processor will get data if re-fired. It keeps the last files that are recent. However as you are in an old version I am not sure what was the design at that version. Please try the cron expression and see if it helps. If you can upgrade at least for NiFi 1.2. Several bugs have been corrected. If you found that this answer addressed your question, please take a moment to click "Accept" below. ... View more

@sri chaturvedi I just tried the following call on my NiFi: Get my root PG ID with the following call : http://localhost:8080/nifi-api/process-groups/root/ (it's c2d938e4-015e-1000-7261-ec94e01a500d) Get all connections of my root PG with their IDs : http://localhost:8080/nifi-api/process-groups/c2d938e4-015e-1000-7261-ec94e01a500d/connections Drop flow files from on of the connection with ID 774b15d2-015f-1000-d65f-951f21fdcf4d with the call http://localhost:8080/nifi-api/flowfile-queues/774b15d2-015f-1000-d65f-951f21fdcf4d/drop-requests You can put this logic inside a script and use Curl to call NiFi API. I hope this helps ... View more

Hi @sri chaturvedi You can right click on connection and select empty queue ... View more

Hi @Tanmoy If you are referring to the lineage duration in NiFi provenance UI (see below pic) then you can use SiteToSiteProvenanceReportingTask to send provenance data to a NiFi cluster. Once NiFi receives it, it can store it it where ever you want (File, index in Solr, Database, etc). To test it, go to hamburger top-right menu, controller settings, reporting tasks and add a S2SProvenanceRT. Configure it to send data to the same cluster like below: Notice the Input Port Name attribute. I called it prov so I need to add an input port add to NiFi flow. Then, from this input port I'll decide what to do with provenance data (store it somewhere). Data you will be receiving looks like this: { "eventId": "5e5acd4a-46e5-4bb7-b957-22b89b7c4bb5", "eventOrdinal": 67, "eventType": "ATTRIBUTES_MODIFIED", "timestampMillis": 1509535134626, "timestamp": "2017-11-01T11:18:54.626Z", "durationMillis": -1, "lineageStart": 1509535134620, "componentId": "3829068d-015f-1000-b540-41c80254f8c7", "componentType": "UpdateAttribute", "componentName": "UpdateAttribute", "entityId": "e72bed81-0c26-4dc8-85e2-b4ac7940fcfe", "entityType": "org.apache.nifi.flowfile.FlowFile", "entitySize": 258, "previousEntitySize": 258, "updatedAttributes": { "project_id": "project_1" }, "previousAttributes": { "path": "./", "uuid": "e72bed81-0c26-4dc8-85e2-b4ac7940fcfe", "filename": "721201919566618" }, "actorHostname": "abdelkrjidjmbp2", "contentURI": "http://abdelkrjidjmbp2:8080/nifi-api/provenance-events/67/content/output", "previousContentURI": "http://abdelkrjidjmbp2:8080/nifi-api/provenance-events/67/content/input", "parentIds": [], "childIds": [], "platform": "nifi", "application": "NiFi Flow" }, You can see the name of the processor (UpdateAttribute), its type (UpdateAttribute), its ID (3829068d-015f-1000-b540-41c80254f8c7) and the flow file ID (5e5acd4a-46e5-4bb7-b957-22b89b7c4bb5). To get the Provenance duration you need to do timestampMillis - lineageStart. In my example it's 1509535134626 - 1509535134620 which is 6 ms like in the first screenshot ... View more