Hi @Timothy Spann, take a look at this answer I just provided for a similar question. Although, you should also look at SiteToSiteProvenanceReportingTask , which allows you "export" them from the NiFi provenance repository and ingest them into a NiFi instance (could be remote, could be same) as pure data and then manipulate them as you would any other data (e.g. feed to PutHDFS ). ... View more

@Greg Keys unfortunately I think you are correct that ExecuteScript is the best way to achieve this right now. As far as I know, the PutFile processor cannot append to an existing file. You are given the option to deal with conflicting files using "replace", "ignore", or "fail" as a resolution strategy. You should submit an Apache Jira to add this functionality. I could see difficulties with file locks and flushing the buffer given the streaming nature of NiFi and I think further investigation is needed. ... View more

You can send a POST request to https://host:port/nifi-api/provenance which will submit a provenance query. The contents of this request should be {"provenance":{"request":{"maxResults":1000}}} (with a configurable count). The response will contain an identifier for the query (as it may take a long time to execute). You can then submit a GET request to https://host:port/nifi-api/provenance/{query-id} which will respond with the results of the query. From this response, you can iterate/extract specific provenance event IDs and request more information using the GET /provenance-events/{id} method. You can also add additional filters and search terms in the initial query to refine it (see GET /provenance/search-options for available options). Example response to search options ( GET https://nifi.nifi.apache.org:9443/nifi-api/provenance/search-options ): {"provenanceOptions":{"searchableFields":[{"id":"EventType","field":"eventType","label":"Event Type","type":"STRING"},{"id":"FlowFileUUID","field":"uuid","label":"FlowFile UUID","type":"STRING"},{"id":"Filename","field":"filename","label":"Filename","type":"STRING"},{"id":"ProcessorID","field":"processorId","label":"Component ID","type":"STRING"},{"id":"Relationship","field":"relationship","label":"Relationship","type":"STRING"}]}} Example response to initial query submission ( POST https://nifi.nifi.apache.org:9443/nifi-api/provenance ): {"provenance":{"id":"0e74ca7f-0158-1000-e780-8ec16cb486ba","uri":"https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba","submissionTime":"10/28/2016 20:21:24.866 PDT","expiration":"10/28/2016 20:51:24.868 PDT","percentCompleted":0,"finished":false,"request":{"searchTerms":{},"maxResults":1000},"results":{"provenanceEvents":[],"total":"0","totalCount":0,"generated":"20:21:24 PDT","oldestEvent":"10/28/2016 20:14:44 PDT","timeOffset":-25200000}}} Example response to query update request ( GET https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba ): {"provenance":{"id":"0e74ca7f-0158-1000-e780-8ec16cb486ba","uri":"https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba","submissionTime":"10/28/2016 20:21:24.866 PDT","expiration":"10/28/2016 20:51:24.876 PDT","percentCompleted":100,"finished":true,"request":{"searchTerms":{},"maxResults":1000},"results":{"provenanceEvents":[{"id":"13","eventId":13,"eventTime":"10/28/2016 20:15:43.606 PDT","lineageDuration":11,"eventType":"DROP","flowFileUuid":"66d1354f-1c0d-4658-9263-17d77ef741df","fileSize":"0 bytes","fileSizeBytes":0,"groupId":"0947f405-0158-1000-d643-3299cb111b40","componentId":"0e6f37b9-0158-1000-254c-d79aeb605761","componentType":"LogAttribute","componentName":"LogAttribute","attributes":[{"name":"filename","value":"1234593201638265","previousValue":"1234593201638265"},{"name":"path","value":"./","previousValue":"./"},{"name":"uuid","value":"66d1354f-1c0d-4658-9263-17d77ef741df","previousValue":"66d1354f-1c0d-4658-9263-17d77ef741df"}],"parentUuids":[],"childUuids":[],"details":"Auto-Terminated by success Relationship","contentEqual":false,"inputContentAvailable":false,"outputContentAvailable":false,"outputContentClaimFileSize":"0 bytes","outputContentClaimFileSizeBytes":0,"replayAvailable":false,"replayExplanation":"Cannot replay data from Provenance Event because the event does not contain the required Content Claim","sourceConnectionIdentifier":"0e6f4ed9-0158-1000-0ab4-97a391fb36b8"}, ... {"id":"0","eventId":0,"eventTime":"10/28/2016 20:14:44.820 PDT","lineageDuration":10,"eventType":"CREATE","flowFileUuid":"3e5d9a82-f185-48d5-b0e4-f3b818f81a13","fileSize":"0 bytes","fileSizeBytes":0,"groupId":"0947f405-0158-1000-d643-3299cb111b40","componentId":"0e6e6df1-0158-1000-e997-ec080b4cdd98","componentType":"GenerateFlowFile","componentName":"GenerateFlowFile","attributes":[{"name":"filename","value":"1234534415887832"},{"name":"path","value":"./"},{"name":"uuid","value":"3e5d9a82-f185-48d5-b0e4-f3b818f81a13"}],"parentUuids":[],"childUuids":[],"contentEqual":false,"inputContentAvailable":false,"outputContentAvailable":false,"outputContentClaimFileSize":"0 bytes","outputContentClaimFileSizeBytes":0,"replayAvailable":false,"replayExplanation":"Cannot replay data from Provenance Event because the event does not contain the required Content Claim"}],"total":"14","totalCount":14,"generated":"20:21:25 PDT","oldestEvent":"10/28/2016 20:14:44 PDT","timeOffset":-25200000}}} Response to query delete request when processing is complete ( DELETE https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba ): {} ... View more