Nifi- how to get provenance event id in nifi?

[#1]

Is there any way to get provenance event id? In nifi-rest api ,I want to get a provenance event use /provenance-events/{id}, What should I do? Thanks.

[#2 · Accepted solution]

You can send a POST request to https://host:port/nifi-api/provenance which will submit a provenance query. The contents of this request should be {"provenance":{"request":{"maxResults":1000}}} (with a configurable count). The response will contain an identifier for the query (as it may take a long time to execute). You can then submit a GET request to https://host:port/nifi-api/provenance/{query-id} which will respond with the results of the query. From this response, you can iterate/extract specific provenance event IDs and request more information using the GET /provenance-events/{id} method. You can also add additional filters and search terms in the initial query to refine it (see GET /provenance/search-options for available options).

Example response to search options ( GET https://nifi.nifi.apache.org:9443/nifi-api/provenance/search-options):

{"provenanceOptions":{"searchableFields":[{"id":"EventType","field":"eventType","label":"Event Type","type":"STRING"},{"id":"FlowFileUUID","field":"uuid","label":"FlowFile UUID","type":"STRING"},{"id":"Filename","field":"filename","label":"Filename","type":"STRING"},{"id":"ProcessorID","field":"processorId","label":"Component ID","type":"STRING"},{"id":"Relationship","field":"relationship","label":"Relationship","type":"STRING"}]}}

Example response to initial query submission ( POST https://nifi.nifi.apache.org:9443/nifi-api/provenance):

{"provenance":{"id":"0e74ca7f-0158-1000-e780-8ec16cb486ba","uri":"https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba","submissionTime":"10/28/2016 20:21:24.866 PDT","expiration":"10/28/2016 20:51:24.868 PDT","percentCompleted":0,"finished":false,"request":{"searchTerms":{},"maxResults":1000},"results":{"provenanceEvents":[],"total":"0","totalCount":0,"generated":"20:21:24 PDT","oldestEvent":"10/28/2016 20:14:44 PDT","timeOffset":-25200000}}}

Example response to query update request ( GET https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba):

{"provenance":{"id":"0e74ca7f-0158-1000-e780-8ec16cb486ba","uri":"https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba","submissionTime":"10/28/2016 20:21:24.866 PDT","expiration":"10/28/2016 20:51:24.876 PDT","percentCompleted":100,"finished":true,"request":{"searchTerms":{},"maxResults":1000},"results":{"provenanceEvents":[{"id":"13","eventId":13,"eventTime":"10/28/2016 20:15:43.606 PDT","lineageDuration":11,"eventType":"DROP","flowFileUuid":"66d1354f-1c0d-4658-9263-17d77ef741df","fileSize":"0 bytes","fileSizeBytes":0,"groupId":"0947f405-0158-1000-d643-3299cb111b40","componentId":"0e6f37b9-0158-1000-254c-d79aeb605761","componentType":"LogAttribute","componentName":"LogAttribute","attributes":[{"name":"filename","value":"1234593201638265","previousValue":"1234593201638265"},{"name":"path","value":"./","previousValue":"./"},{"name":"uuid","value":"66d1354f-1c0d-4658-9263-17d77ef741df","previousValue":"66d1354f-1c0d-4658-9263-17d77ef741df"}],"parentUuids":[],"childUuids":[],"details":"Auto-Terminated by success Relationship","contentEqual":false,"inputContentAvailable":false,"outputContentAvailable":false,"outputContentClaimFileSize":"0 bytes","outputContentClaimFileSizeBytes":0,"replayAvailable":false,"replayExplanation":"Cannot replay data from Provenance Event because the event does not contain the required Content Claim","sourceConnectionIdentifier":"0e6f4ed9-0158-1000-0ab4-97a391fb36b8"}, 

... 

{"id":"0","eventId":0,"eventTime":"10/28/2016 20:14:44.820 PDT","lineageDuration":10,"eventType":"CREATE","flowFileUuid":"3e5d9a82-f185-48d5-b0e4-f3b818f81a13","fileSize":"0 bytes","fileSizeBytes":0,"groupId":"0947f405-0158-1000-d643-3299cb111b40","componentId":"0e6e6df1-0158-1000-e997-ec080b4cdd98","componentType":"GenerateFlowFile","componentName":"GenerateFlowFile","attributes":[{"name":"filename","value":"1234534415887832"},{"name":"path","value":"./"},{"name":"uuid","value":"3e5d9a82-f185-48d5-b0e4-f3b818f81a13"}],"parentUuids":[],"childUuids":[],"contentEqual":false,"inputContentAvailable":false,"outputContentAvailable":false,"outputContentClaimFileSize":"0 bytes","outputContentClaimFileSizeBytes":0,"replayAvailable":false,"replayExplanation":"Cannot replay data from Provenance Event because the event does not contain the required Content Claim"}],"total":"14","totalCount":14,"generated":"20:21:25 PDT","oldestEvent":"10/28/2016 20:14:44 PDT","timeOffset":-25200000}}}

Response to query delete request when processing is complete ( DELETE https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba):

{}

[#3]

Thanks a lot ,I've managed to solve this problem with your help

[#4]

Hi @m mary / @Andy LoPresto: As mentioned above i got provenance id(ex id : 0e74ca7f-0158-1000-e780-8ec16cb486ba). Now i want to run "/provenance-events/{id}" rest API. So i have replced above id with this rest API(/provenance-events/0e74ca7f-0158-1000-e780-8ec16cb486ba). But it is throwing "HTTP/1.1 404 Not Found" error. Please help me which id I need to replace here to get the provenance-events details.

[#5]

hi. provenance id is changing.every id can be used once,you can't use the provenance id to get "/provenance-events/{id}" again,you can get id and get the url directly.

[#6]

Hi, As per your suggestion I have replaced above rest API (/provenance-events/{id}) ID with UUID/Component Id/Identifier. Still it is throwing "404 Not Found" error.

[#7]

Can you please post the complete output of curl or whatever REST client you are using? The complete URI to use is http(s)://host:port/nifi-api/provenance-events/{id}.

[#8]

@Andy LoPresto

I got below ouput after executing GET request to http://<hostname>:9090/nifi-api/provenance/query-id :

{"provenanceEvents":[{"id":"a9d352d4-20e9-4fbb-851c-3191204d58ba4","eventId":4,"eventTime":"04/18/2017 09:30:14.775 UTC","lineageDuration":1,"eventType":"CREATE","flowFileUuid":"c87d810d-17b7-4c5f-b313-dd4653e81a8d","fileSize":"0 bytes","fileSizeBytes":0,"clusterNodeId":"a9d352d4-20e9-4fbb-851c-3191204d58ba","clusterNodeAddress":"amnifi1.openstacklocal:9090","groupId":"b90e8d65-015a-1000-22ce-e993c787469e","componentId":"61665eb8-015b-1000-0000-0000653a830f","componentType":"ListFTP","componentName":"AAAAAAAAAAAA","attributes":[{"name":"file.group","value":"0"},{"name":"file.lastModifiedTime","value":"2017-04-18T09:30:00+0000"},{"name":"file.owner","value":"0"},{"name":"file.permissions","value":"rw-r--r--"},{"name":"filename","value":"jj1.txt"},{"name":"ftp.listing.user","value":"akshay"},{"name":"ftp.remote.host","value":"amnifi3.openstacklocal"},{"name":"ftp.remote.port","value":"21"},{"name":"path","value":"ALL_DAY_WORK"},{"name":"uuid","value":"c87d810d-17b7-4c5f-b313-dd4653e81a8d"}],"parentUuids":[],"childUuids":[],"contentEqual":false,"inputContentAvailable":false,"outputContentAvailable":false,"outputContentClaimFileSize":"0 bytes","outputContentClaimFileSizeBytes":0,"replayAvailable":false,"replayExplanation":"Cannot replay data from Provenance Event because the event does not contain the required Content Claim”},

Now when I am trying to submit GET request to get more info about provenance event I am getting below error message :

$ curl -i -X GET http://<hostname>:9090/nifi-api/provenance-events/{4}

HTTP/1.1 400 Bad Request
Date: Wed, 19 Apr 2017 07:21:13 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Server: Jetty(9.3.9.v20160517)
The cluster node identifier must be specified.

Nifi is clustered. Not sure where I should mentioned cluster node id. Please help.

[#9]

As noted in the REST API, clusterNodeId is a required field in the GET /provenance-events/{id} request when running in clustered mode. Use the Developer Tools of your browser to watch the requests that the UI fires when you get the details of a provenance event and you should see where to include the clusterNodeId in the request (my instance is not running in a cluster so it is not present in my request).

[#10]

Please refer below link in order to get more information about a specific provenance event for nifi which is running in clustered mode : - How to get information of a specific provenance event when nifi is running in standalone/clustered...