Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Nifi- how to get provenance event id in nifi?

avatar
Contributor

Is there any way to get provenance event id? In nifi-rest api ,I want to get a provenance event use /provenance-events/{id}, What should I do? Thanks.

1 ACCEPTED SOLUTION

avatar

You can send a POST request to https://host:port/nifi-api/provenance which will submit a provenance query. The contents of this request should be {"provenance":{"request":{"maxResults":1000}}} (with a configurable count). The response will contain an identifier for the query (as it may take a long time to execute). You can then submit a GET request to https://host:port/nifi-api/provenance/{query-id} which will respond with the results of the query. From this response, you can iterate/extract specific provenance event IDs and request more information using the GET /provenance-events/{id} method. You can also add additional filters and search terms in the initial query to refine it (see GET /provenance/search-options for available options).

Example response to search options ( GET https://nifi.nifi.apache.org:9443/nifi-api/provenance/search-options):

{"provenanceOptions":{"searchableFields":[{"id":"EventType","field":"eventType","label":"Event Type","type":"STRING"},{"id":"FlowFileUUID","field":"uuid","label":"FlowFile UUID","type":"STRING"},{"id":"Filename","field":"filename","label":"Filename","type":"STRING"},{"id":"ProcessorID","field":"processorId","label":"Component ID","type":"STRING"},{"id":"Relationship","field":"relationship","label":"Relationship","type":"STRING"}]}}

Example response to initial query submission ( POST https://nifi.nifi.apache.org:9443/nifi-api/provenance):

{"provenance":{"id":"0e74ca7f-0158-1000-e780-8ec16cb486ba","uri":"https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba","submissionTime":"10/28/2016 20:21:24.866 PDT","expiration":"10/28/2016 20:51:24.868 PDT","percentCompleted":0,"finished":false,"request":{"searchTerms":{},"maxResults":1000},"results":{"provenanceEvents":[],"total":"0","totalCount":0,"generated":"20:21:24 PDT","oldestEvent":"10/28/2016 20:14:44 PDT","timeOffset":-25200000}}}

Example response to query update request ( GET https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba):

{"provenance":{"id":"0e74ca7f-0158-1000-e780-8ec16cb486ba","uri":"https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba","submissionTime":"10/28/2016 20:21:24.866 PDT","expiration":"10/28/2016 20:51:24.876 PDT","percentCompleted":100,"finished":true,"request":{"searchTerms":{},"maxResults":1000},"results":{"provenanceEvents":[{"id":"13","eventId":13,"eventTime":"10/28/2016 20:15:43.606 PDT","lineageDuration":11,"eventType":"DROP","flowFileUuid":"66d1354f-1c0d-4658-9263-17d77ef741df","fileSize":"0 bytes","fileSizeBytes":0,"groupId":"0947f405-0158-1000-d643-3299cb111b40","componentId":"0e6f37b9-0158-1000-254c-d79aeb605761","componentType":"LogAttribute","componentName":"LogAttribute","attributes":[{"name":"filename","value":"1234593201638265","previousValue":"1234593201638265"},{"name":"path","value":"./","previousValue":"./"},{"name":"uuid","value":"66d1354f-1c0d-4658-9263-17d77ef741df","previousValue":"66d1354f-1c0d-4658-9263-17d77ef741df"}],"parentUuids":[],"childUuids":[],"details":"Auto-Terminated by success Relationship","contentEqual":false,"inputContentAvailable":false,"outputContentAvailable":false,"outputContentClaimFileSize":"0 bytes","outputContentClaimFileSizeBytes":0,"replayAvailable":false,"replayExplanation":"Cannot replay data from Provenance Event because the event does not contain the required Content Claim","sourceConnectionIdentifier":"0e6f4ed9-0158-1000-0ab4-97a391fb36b8"}, 

... 

{"id":"0","eventId":0,"eventTime":"10/28/2016 20:14:44.820 PDT","lineageDuration":10,"eventType":"CREATE","flowFileUuid":"3e5d9a82-f185-48d5-b0e4-f3b818f81a13","fileSize":"0 bytes","fileSizeBytes":0,"groupId":"0947f405-0158-1000-d643-3299cb111b40","componentId":"0e6e6df1-0158-1000-e997-ec080b4cdd98","componentType":"GenerateFlowFile","componentName":"GenerateFlowFile","attributes":[{"name":"filename","value":"1234534415887832"},{"name":"path","value":"./"},{"name":"uuid","value":"3e5d9a82-f185-48d5-b0e4-f3b818f81a13"}],"parentUuids":[],"childUuids":[],"contentEqual":false,"inputContentAvailable":false,"outputContentAvailable":false,"outputContentClaimFileSize":"0 bytes","outputContentClaimFileSizeBytes":0,"replayAvailable":false,"replayExplanation":"Cannot replay data from Provenance Event because the event does not contain the required Content Claim"}],"total":"14","totalCount":14,"generated":"20:21:25 PDT","oldestEvent":"10/28/2016 20:14:44 PDT","timeOffset":-25200000}}}

Response to query delete request when processing is complete ( DELETE https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba):

{}

View solution in original post

9 REPLIES 9

avatar

You can send a POST request to https://host:port/nifi-api/provenance which will submit a provenance query. The contents of this request should be {"provenance":{"request":{"maxResults":1000}}} (with a configurable count). The response will contain an identifier for the query (as it may take a long time to execute). You can then submit a GET request to https://host:port/nifi-api/provenance/{query-id} which will respond with the results of the query. From this response, you can iterate/extract specific provenance event IDs and request more information using the GET /provenance-events/{id} method. You can also add additional filters and search terms in the initial query to refine it (see GET /provenance/search-options for available options).

Example response to search options ( GET https://nifi.nifi.apache.org:9443/nifi-api/provenance/search-options):

{"provenanceOptions":{"searchableFields":[{"id":"EventType","field":"eventType","label":"Event Type","type":"STRING"},{"id":"FlowFileUUID","field":"uuid","label":"FlowFile UUID","type":"STRING"},{"id":"Filename","field":"filename","label":"Filename","type":"STRING"},{"id":"ProcessorID","field":"processorId","label":"Component ID","type":"STRING"},{"id":"Relationship","field":"relationship","label":"Relationship","type":"STRING"}]}}

Example response to initial query submission ( POST https://nifi.nifi.apache.org:9443/nifi-api/provenance):

{"provenance":{"id":"0e74ca7f-0158-1000-e780-8ec16cb486ba","uri":"https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba","submissionTime":"10/28/2016 20:21:24.866 PDT","expiration":"10/28/2016 20:51:24.868 PDT","percentCompleted":0,"finished":false,"request":{"searchTerms":{},"maxResults":1000},"results":{"provenanceEvents":[],"total":"0","totalCount":0,"generated":"20:21:24 PDT","oldestEvent":"10/28/2016 20:14:44 PDT","timeOffset":-25200000}}}

Example response to query update request ( GET https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba):

{"provenance":{"id":"0e74ca7f-0158-1000-e780-8ec16cb486ba","uri":"https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba","submissionTime":"10/28/2016 20:21:24.866 PDT","expiration":"10/28/2016 20:51:24.876 PDT","percentCompleted":100,"finished":true,"request":{"searchTerms":{},"maxResults":1000},"results":{"provenanceEvents":[{"id":"13","eventId":13,"eventTime":"10/28/2016 20:15:43.606 PDT","lineageDuration":11,"eventType":"DROP","flowFileUuid":"66d1354f-1c0d-4658-9263-17d77ef741df","fileSize":"0 bytes","fileSizeBytes":0,"groupId":"0947f405-0158-1000-d643-3299cb111b40","componentId":"0e6f37b9-0158-1000-254c-d79aeb605761","componentType":"LogAttribute","componentName":"LogAttribute","attributes":[{"name":"filename","value":"1234593201638265","previousValue":"1234593201638265"},{"name":"path","value":"./","previousValue":"./"},{"name":"uuid","value":"66d1354f-1c0d-4658-9263-17d77ef741df","previousValue":"66d1354f-1c0d-4658-9263-17d77ef741df"}],"parentUuids":[],"childUuids":[],"details":"Auto-Terminated by success Relationship","contentEqual":false,"inputContentAvailable":false,"outputContentAvailable":false,"outputContentClaimFileSize":"0 bytes","outputContentClaimFileSizeBytes":0,"replayAvailable":false,"replayExplanation":"Cannot replay data from Provenance Event because the event does not contain the required Content Claim","sourceConnectionIdentifier":"0e6f4ed9-0158-1000-0ab4-97a391fb36b8"}, 

... 

{"id":"0","eventId":0,"eventTime":"10/28/2016 20:14:44.820 PDT","lineageDuration":10,"eventType":"CREATE","flowFileUuid":"3e5d9a82-f185-48d5-b0e4-f3b818f81a13","fileSize":"0 bytes","fileSizeBytes":0,"groupId":"0947f405-0158-1000-d643-3299cb111b40","componentId":"0e6e6df1-0158-1000-e997-ec080b4cdd98","componentType":"GenerateFlowFile","componentName":"GenerateFlowFile","attributes":[{"name":"filename","value":"1234534415887832"},{"name":"path","value":"./"},{"name":"uuid","value":"3e5d9a82-f185-48d5-b0e4-f3b818f81a13"}],"parentUuids":[],"childUuids":[],"contentEqual":false,"inputContentAvailable":false,"outputContentAvailable":false,"outputContentClaimFileSize":"0 bytes","outputContentClaimFileSizeBytes":0,"replayAvailable":false,"replayExplanation":"Cannot replay data from Provenance Event because the event does not contain the required Content Claim"}],"total":"14","totalCount":14,"generated":"20:21:25 PDT","oldestEvent":"10/28/2016 20:14:44 PDT","timeOffset":-25200000}}}

Response to query delete request when processing is complete ( DELETE https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba):

{}

avatar
Contributor

Thanks a lot ,I've managed to solve this problem with your help

avatar

Hi @m mary / @Andy LoPresto: As mentioned above i got provenance id(ex id : 0e74ca7f-0158-1000-e780-8ec16cb486ba). Now i want to run "/provenance-events/{id}" rest API. So i have replced above id with this rest API(/provenance-events/0e74ca7f-0158-1000-e780-8ec16cb486ba). But it is throwing "HTTP/1.1 404 Not Found" error. Please help me which id I need to replace here to get the provenance-events details.

avatar
Contributor

hi. provenance id is changing.every id can be used once,you can't use the provenance id to get "/provenance-events/{id}" again,you can get id and get the url directly.

avatar

Hi, As per your suggestion I have replaced above rest API (/provenance-events/{id}) ID with UUID/Component Id/Identifier. Still it is throwing "404 Not Found" error.

avatar

Can you please post the complete output of curl or whatever REST client you are using? The complete URI to use is http(s)://host:port/nifi-api/provenance-events/{id}.

avatar
Rising Star

@Andy LoPresto

I got below ouput after executing GET request to http://<hostname>:9090/nifi-api/provenance/query-id :

{"provenanceEvents":[{"id":"a9d352d4-20e9-4fbb-851c-3191204d58ba4","eventId":4,"eventTime":"04/18/2017 09:30:14.775 UTC","lineageDuration":1,"eventType":"CREATE","flowFileUuid":"c87d810d-17b7-4c5f-b313-dd4653e81a8d","fileSize":"0 bytes","fileSizeBytes":0,"clusterNodeId":"a9d352d4-20e9-4fbb-851c-3191204d58ba","clusterNodeAddress":"amnifi1.openstacklocal:9090","groupId":"b90e8d65-015a-1000-22ce-e993c787469e","componentId":"61665eb8-015b-1000-0000-0000653a830f","componentType":"ListFTP","componentName":"AAAAAAAAAAAA","attributes":[{"name":"file.group","value":"0"},{"name":"file.lastModifiedTime","value":"2017-04-18T09:30:00+0000"},{"name":"file.owner","value":"0"},{"name":"file.permissions","value":"rw-r--r--"},{"name":"filename","value":"jj1.txt"},{"name":"ftp.listing.user","value":"akshay"},{"name":"ftp.remote.host","value":"amnifi3.openstacklocal"},{"name":"ftp.remote.port","value":"21"},{"name":"path","value":"ALL_DAY_WORK"},{"name":"uuid","value":"c87d810d-17b7-4c5f-b313-dd4653e81a8d"}],"parentUuids":[],"childUuids":[],"contentEqual":false,"inputContentAvailable":false,"outputContentAvailable":false,"outputContentClaimFileSize":"0 bytes","outputContentClaimFileSizeBytes":0,"replayAvailable":false,"replayExplanation":"Cannot replay data from Provenance Event because the event does not contain the required Content Claim”},

Now when I am trying to submit GET request to get more info about provenance event I am getting below error message :

$ curl -i -X GET http://<hostname>:9090/nifi-api/provenance-events/{4}

HTTP/1.1 400 Bad Request
Date: Wed, 19 Apr 2017 07:21:13 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Server: Jetty(9.3.9.v20160517)
The cluster node identifier must be specified.

Nifi is clustered. Not sure where I should mentioned cluster node id. Please help.

avatar

As noted in the REST API, clusterNodeId is a required field in the GET /provenance-events/{id} request when running in clustered mode. Use the Developer Tools of your browser to watch the requests that the UI fires when you get the details of a provenance event and you should see where to include the clusterNodeId in the request (my instance is not running in a cluster so it is not present in my request).

14761-screen-shot-2017-04-19-at-102930-am.png

avatar
Rising Star

Please refer below link in order to get more information about a specific provenance event for nifi which is running in clustered mode : - How to get information of a specific provenance event when nifi is running in standalone/clustered...