Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

can we list flow files in nifi server by their duration

Highlighted

can we list flow files in nifi server by their duration

New Contributor

I was looking if we an api or some feature, that list files older than x days for a nifi url and corresponding queue id.

1 REPLY 1
Highlighted

Re: can we list flow files in nifi server by their duration

Master Guru

@NY 

 

Anything you can do via the NiFi canvas, you can also accomplish via the rest-api.

So you can create a queue listing via a rest-api call:

curl http://<nifi-hostname>:<port>/nifi-api/flowfile-queues/<connection-uuid>/listing-requests

The above call will return a response that includes the URL you need to use to retrieve those results.

for example:

http://<nifi-hostname>:<port>/nifi-api/flowfile-queues/<connection-uuid>/listing-requests/1d98d557-0171-1000-ffff-ffffd559ca47


Then query for the listings results

curl http://<nifi-hostname>:<port>/nifi-api/flowfile-queues/<connection-uuid>/listing-requests/1d98d557-0171-1000-ffff-ffffd559ca47

This return a json with all the FlowFiles from that specific connection queue from all nodes in your NiFi cluster.  That json would need to be parsed for info like nodes where the "lineageDuration" epoch time is x amount of time older then now, the "clusterNodeAddress" (which node holds the file), and maybe filename".


and then delete the queue listing when done. (this is important or it stays around using heap space).

curl -X DELETE http://<nifi-hostname>:<port>/nifi-api/flowfile-queues/<connection-uuid>/listing-requests/1d98d557-0171-1000-ffff-ffffd559ca47


Hope this helps,
Matt

Don't have an account?
Coming from Hortonworks? Activate your account here