Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi- how to get provenance event id in nifi?

Solved Go to solution

Nifi- how to get provenance event id in nifi?

New Contributor

Is there any way to get provenance event id? In nifi-rest api ,I want to get a provenance event use /provenance-events/{id}, What should I do? Thanks.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Nifi- how to get provenance event id in nifi?

You can send a POST request to https://host:port/nifi-api/provenance which will submit a provenance query. The contents of this request should be {"provenance":{"request":{"maxResults":1000}}} (with a configurable count). The response will contain an identifier for the query (as it may take a long time to execute). You can then submit a GET request to https://host:port/nifi-api/provenance/{query-id} which will respond with the results of the query. From this response, you can iterate/extract specific provenance event IDs and request more information using the GET /provenance-events/{id} method. You can also add additional filters and search terms in the initial query to refine it (see GET /provenance/search-options for available options).

Example response to search options ( GET https://nifi.nifi.apache.org:9443/nifi-api/provenance/search-options):

{"provenanceOptions":{"searchableFields":[{"id":"EventType","field":"eventType","label":"Event Type","type":"STRING"},{"id":"FlowFileUUID","field":"uuid","label":"FlowFile UUID","type":"STRING"},{"id":"Filename","field":"filename","label":"Filename","type":"STRING"},{"id":"ProcessorID","field":"processorId","label":"Component ID","type":"STRING"},{"id":"Relationship","field":"relationship","label":"Relationship","type":"STRING"}]}}

Example response to initial query submission ( POST https://nifi.nifi.apache.org:9443/nifi-api/provenance):

{"provenance":{"id":"0e74ca7f-0158-1000-e780-8ec16cb486ba","uri":"https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba","submissionTime":"10/28/2016 20:21:24.866 PDT","expiration":"10/28/2016 20:51:24.868 PDT","percentCompleted":0,"finished":false,"request":{"searchTerms":{},"maxResults":1000},"results":{"provenanceEvents":[],"total":"0","totalCount":0,"generated":"20:21:24 PDT","oldestEvent":"10/28/2016 20:14:44 PDT","timeOffset":-25200000}}}

Example response to query update request ( GET https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba):

{"provenance":{"id":"0e74ca7f-0158-1000-e780-8ec16cb486ba","uri":"https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba","submissionTime":"10/28/2016 20:21:24.866 PDT","expiration":"10/28/2016 20:51:24.876 PDT","percentCompleted":100,"finished":true,"request":{"searchTerms":{},"maxResults":1000},"results":{"provenanceEvents":[{"id":"13","eventId":13,"eventTime":"10/28/2016 20:15:43.606 PDT","lineageDuration":11,"eventType":"DROP","flowFileUuid":"66d1354f-1c0d-4658-9263-17d77ef741df","fileSize":"0 bytes","fileSizeBytes":0,"groupId":"0947f405-0158-1000-d643-3299cb111b40","componentId":"0e6f37b9-0158-1000-254c-d79aeb605761","componentType":"LogAttribute","componentName":"LogAttribute","attributes":[{"name":"filename","value":"1234593201638265","previousValue":"1234593201638265"},{"name":"path","value":"./","previousValue":"./"},{"name":"uuid","value":"66d1354f-1c0d-4658-9263-17d77ef741df","previousValue":"66d1354f-1c0d-4658-9263-17d77ef741df"}],"parentUuids":[],"childUuids":[],"details":"Auto-Terminated by success Relationship","contentEqual":false,"inputContentAvailable":false,"outputContentAvailable":false,"outputContentClaimFileSize":"0 bytes","outputContentClaimFileSizeBytes":0,"replayAvailable":false,"replayExplanation":"Cannot replay data from Provenance Event because the event does not contain the required Content Claim","sourceConnectionIdentifier":"0e6f4ed9-0158-1000-0ab4-97a391fb36b8"}, 

... 

{"id":"0","eventId":0,"eventTime":"10/28/2016 20:14:44.820 PDT","lineageDuration":10,"eventType":"CREATE","flowFileUuid":"3e5d9a82-f185-48d5-b0e4-f3b818f81a13","fileSize":"0 bytes","fileSizeBytes":0,"groupId":"0947f405-0158-1000-d643-3299cb111b40","componentId":"0e6e6df1-0158-1000-e997-ec080b4cdd98","componentType":"GenerateFlowFile","componentName":"GenerateFlowFile","attributes":[{"name":"filename","value":"1234534415887832"},{"name":"path","value":"./"},{"name":"uuid","value":"3e5d9a82-f185-48d5-b0e4-f3b818f81a13"}],"parentUuids":[],"childUuids":[],"contentEqual":false,"inputContentAvailable":false,"outputContentAvailable":false,"outputContentClaimFileSize":"0 bytes","outputContentClaimFileSizeBytes":0,"replayAvailable":false,"replayExplanation":"Cannot replay data from Provenance Event because the event does not contain the required Content Claim"}],"total":"14","totalCount":14,"generated":"20:21:25 PDT","oldestEvent":"10/28/2016 20:14:44 PDT","timeOffset":-25200000}}}

Response to query delete request when processing is complete ( DELETE https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba):

{}
9 REPLIES 9

Re: Nifi- how to get provenance event id in nifi?

You can send a POST request to https://host:port/nifi-api/provenance which will submit a provenance query. The contents of this request should be {"provenance":{"request":{"maxResults":1000}}} (with a configurable count). The response will contain an identifier for the query (as it may take a long time to execute). You can then submit a GET request to https://host:port/nifi-api/provenance/{query-id} which will respond with the results of the query. From this response, you can iterate/extract specific provenance event IDs and request more information using the GET /provenance-events/{id} method. You can also add additional filters and search terms in the initial query to refine it (see GET /provenance/search-options for available options).

Example response to search options ( GET https://nifi.nifi.apache.org:9443/nifi-api/provenance/search-options):

{"provenanceOptions":{"searchableFields":[{"id":"EventType","field":"eventType","label":"Event Type","type":"STRING"},{"id":"FlowFileUUID","field":"uuid","label":"FlowFile UUID","type":"STRING"},{"id":"Filename","field":"filename","label":"Filename","type":"STRING"},{"id":"ProcessorID","field":"processorId","label":"Component ID","type":"STRING"},{"id":"Relationship","field":"relationship","label":"Relationship","type":"STRING"}]}}

Example response to initial query submission ( POST https://nifi.nifi.apache.org:9443/nifi-api/provenance):

{"provenance":{"id":"0e74ca7f-0158-1000-e780-8ec16cb486ba","uri":"https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba","submissionTime":"10/28/2016 20:21:24.866 PDT","expiration":"10/28/2016 20:51:24.868 PDT","percentCompleted":0,"finished":false,"request":{"searchTerms":{},"maxResults":1000},"results":{"provenanceEvents":[],"total":"0","totalCount":0,"generated":"20:21:24 PDT","oldestEvent":"10/28/2016 20:14:44 PDT","timeOffset":-25200000}}}

Example response to query update request ( GET https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba):

{"provenance":{"id":"0e74ca7f-0158-1000-e780-8ec16cb486ba","uri":"https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba","submissionTime":"10/28/2016 20:21:24.866 PDT","expiration":"10/28/2016 20:51:24.876 PDT","percentCompleted":100,"finished":true,"request":{"searchTerms":{},"maxResults":1000},"results":{"provenanceEvents":[{"id":"13","eventId":13,"eventTime":"10/28/2016 20:15:43.606 PDT","lineageDuration":11,"eventType":"DROP","flowFileUuid":"66d1354f-1c0d-4658-9263-17d77ef741df","fileSize":"0 bytes","fileSizeBytes":0,"groupId":"0947f405-0158-1000-d643-3299cb111b40","componentId":"0e6f37b9-0158-1000-254c-d79aeb605761","componentType":"LogAttribute","componentName":"LogAttribute","attributes":[{"name":"filename","value":"1234593201638265","previousValue":"1234593201638265"},{"name":"path","value":"./","previousValue":"./"},{"name":"uuid","value":"66d1354f-1c0d-4658-9263-17d77ef741df","previousValue":"66d1354f-1c0d-4658-9263-17d77ef741df"}],"parentUuids":[],"childUuids":[],"details":"Auto-Terminated by success Relationship","contentEqual":false,"inputContentAvailable":false,"outputContentAvailable":false,"outputContentClaimFileSize":"0 bytes","outputContentClaimFileSizeBytes":0,"replayAvailable":false,"replayExplanation":"Cannot replay data from Provenance Event because the event does not contain the required Content Claim","sourceConnectionIdentifier":"0e6f4ed9-0158-1000-0ab4-97a391fb36b8"}, 

... 

{"id":"0","eventId":0,"eventTime":"10/28/2016 20:14:44.820 PDT","lineageDuration":10,"eventType":"CREATE","flowFileUuid":"3e5d9a82-f185-48d5-b0e4-f3b818f81a13","fileSize":"0 bytes","fileSizeBytes":0,"groupId":"0947f405-0158-1000-d643-3299cb111b40","componentId":"0e6e6df1-0158-1000-e997-ec080b4cdd98","componentType":"GenerateFlowFile","componentName":"GenerateFlowFile","attributes":[{"name":"filename","value":"1234534415887832"},{"name":"path","value":"./"},{"name":"uuid","value":"3e5d9a82-f185-48d5-b0e4-f3b818f81a13"}],"parentUuids":[],"childUuids":[],"contentEqual":false,"inputContentAvailable":false,"outputContentAvailable":false,"outputContentClaimFileSize":"0 bytes","outputContentClaimFileSizeBytes":0,"replayAvailable":false,"replayExplanation":"Cannot replay data from Provenance Event because the event does not contain the required Content Claim"}],"total":"14","totalCount":14,"generated":"20:21:25 PDT","oldestEvent":"10/28/2016 20:14:44 PDT","timeOffset":-25200000}}}

Response to query delete request when processing is complete ( DELETE https://nifi.nifi.apache.org:9443/nifi-api/provenance/0e74ca7f-0158-1000-e780-8ec16cb486ba):

{}

Re: Nifi- how to get provenance event id in nifi?

New Contributor

Thanks a lot ,I've managed to solve this problem with your help

Re: Nifi- how to get provenance event id in nifi?

New Contributor

Hi @m mary / @Andy LoPresto: As mentioned above i got provenance id(ex id : 0e74ca7f-0158-1000-e780-8ec16cb486ba). Now i want to run "/provenance-events/{id}" rest API. So i have replced above id with this rest API(/provenance-events/0e74ca7f-0158-1000-e780-8ec16cb486ba). But it is throwing "HTTP/1.1 404 Not Found" error. Please help me which id I need to replace here to get the provenance-events details.

Re: Nifi- how to get provenance event id in nifi?

New Contributor

hi. provenance id is changing.every id can be used once,you can't use the provenance id to get "/provenance-events/{id}" again,you can get id and get the url directly.

Re: Nifi- how to get provenance event id in nifi?

New Contributor

Hi, As per your suggestion I have replaced above rest API (/provenance-events/{id}) ID with UUID/Component Id/Identifier. Still it is throwing "404 Not Found" error.

Re: Nifi- how to get provenance event id in nifi?

Can you please post the complete output of curl or whatever REST client you are using? The complete URI to use is http(s)://host:port/nifi-api/provenance-events/{id}.

Re: Nifi- how to get provenance event id in nifi?

Contributor

@Andy LoPresto

I got below ouput after executing GET request to http://<hostname>:9090/nifi-api/provenance/query-id :

{"provenanceEvents":[{"id":"a9d352d4-20e9-4fbb-851c-3191204d58ba4","eventId":4,"eventTime":"04/18/2017 09:30:14.775 UTC","lineageDuration":1,"eventType":"CREATE","flowFileUuid":"c87d810d-17b7-4c5f-b313-dd4653e81a8d","fileSize":"0 bytes","fileSizeBytes":0,"clusterNodeId":"a9d352d4-20e9-4fbb-851c-3191204d58ba","clusterNodeAddress":"amnifi1.openstacklocal:9090","groupId":"b90e8d65-015a-1000-22ce-e993c787469e","componentId":"61665eb8-015b-1000-0000-0000653a830f","componentType":"ListFTP","componentName":"AAAAAAAAAAAA","attributes":[{"name":"file.group","value":"0"},{"name":"file.lastModifiedTime","value":"2017-04-18T09:30:00+0000"},{"name":"file.owner","value":"0"},{"name":"file.permissions","value":"rw-r--r--"},{"name":"filename","value":"jj1.txt"},{"name":"ftp.listing.user","value":"akshay"},{"name":"ftp.remote.host","value":"amnifi3.openstacklocal"},{"name":"ftp.remote.port","value":"21"},{"name":"path","value":"ALL_DAY_WORK"},{"name":"uuid","value":"c87d810d-17b7-4c5f-b313-dd4653e81a8d"}],"parentUuids":[],"childUuids":[],"contentEqual":false,"inputContentAvailable":false,"outputContentAvailable":false,"outputContentClaimFileSize":"0 bytes","outputContentClaimFileSizeBytes":0,"replayAvailable":false,"replayExplanation":"Cannot replay data from Provenance Event because the event does not contain the required Content Claim”},

Now when I am trying to submit GET request to get more info about provenance event I am getting below error message :

$ curl -i -X GET http://<hostname>:9090/nifi-api/provenance-events/{4}

HTTP/1.1 400 Bad Request
Date: Wed, 19 Apr 2017 07:21:13 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Server: Jetty(9.3.9.v20160517)
The cluster node identifier must be specified.

Nifi is clustered. Not sure where I should mentioned cluster node id. Please help.

Re: Nifi- how to get provenance event id in nifi?

As noted in the REST API, clusterNodeId is a required field in the GET /provenance-events/{id} request when running in clustered mode. Use the Developer Tools of your browser to watch the requests that the UI fires when you get the details of a provenance event and you should see where to include the clusterNodeId in the request (my instance is not running in a cluster so it is not present in my request).

14761-screen-shot-2017-04-19-at-102930-am.png

Highlighted

Re: Nifi- how to get provenance event id in nifi?

Contributor

Please refer below link in order to get more information about a specific provenance event for nifi which is running in clustered mode : - How to get information of a specific provenance event when nifi is running in standalone/clustered...

Don't have an account?
Coming from Hortonworks? Activate your account here