[NIFI-API] Why only one node send the result of data provenance

[#1]

Hi all,

I Have a cluster NIFI with 3 node, when I'm making GET dataprovenance with python script, only nifi-node003 sent me the result.

So can't receive dataprovenance from nifi-node001 and nifi-node002

What's wrong ?

thanks for helps

"{\"provenance\":{\"request\":{\"maxResults\":1000,\"startDate\":\"08/23/2017 03:00:00 CEST\",\"endDate\":\"08/23/2017 03:59:59 CEST\",\"searchTerms\":{\"EventType\":\"SEND\",\"ProcessorID\":\"0dd63d71-8620-1ce1-be9d-587c5f6ec679\"}}}}"

    conn=httplib.HTTPSConnection(clusterNode, 9443, key_file=None, cert_file="/var/opt/hosting/nifi/conf/nifi-001.pem")

    headers = {"Content-Type": "application/json", "Accept": "application/json, text/javascript"}
    conn.request("POST", "/nifi-api/provenance", json.dumps(params), headers)
    response = conn.getresponse()
    result = json.loads(response.read())
    conn.close()

    conn.request("GET", "/nifi-api/provenance/" + result['provenance']['id'])

    responseGet = conn.getresponse()
    resultGet = json.loads(responseGet.read())

    conn.request("DELETE", "/nifi-api/provenance/" + resultGet['provenance']['id'])
    res = conn.getresponse()
    conn.close()

[#2]

Hi @mayki wogno,

The POST /provenance endpoint submits a provenance request. The GET /provenance/{id} endpoint retrieves the provenance request object, and the clusterNodeId is the id of the cluster node where the request exists (ie, the node doing the aggregation processing).

Does that answer your question?

[#3]

@kdoran:

With the web-ui I got 18 result with my search data provenance, but when i'll doing with python script i got only 5 (only has the result from node003)

[#4]

Hi @mayki wogno. Can you post the API call you make and the full JSON response?

[#5]

@mayki wogno

My guess here is that your API call is only directed at nifi003.

When you run a query from within the UI, your query request is replicated all nodes. The results from each of those individual queries are returned to the node your browser is currently displaying. The results are correlated in the provenance results UI you see above.

You can use the developer tools in your browser to see all the rest-api calls that are made when you do your provence query via the UI. You will need to make sure you make all the same queries when interacting with the nodes rest-api directly.

Thanks,

Matt

[#6]

@matt, by default the api get a query from cluster except if we use clusternodeid in the parameter of query, right ?