NiFi: Provenance list capture

[#1]

Hi,

I wanted to be able to track how many message (files) were consumed by the ConsumeKafka process and size of it, for a particular time window. With reference to that, please advise on the below:

1) Can we be able to find the list of items (see attached, such as file size, queued duration) that got processed, even when they are no longer in the queue?

2) Can we query items in queue specific to a particular processor?

3) Is it possible to retrieve and post the above to a custom Amabari metrics?

The objective is to maintain a trend of the average message size and the number of messages, specific to a particular topic (configured in the ConsumeKafka processor).

[#2 · Accepted solution]

There is both a push (reporting api) and pull (restful api) way to automate metrics collection in NiFI. See this post for an overview: https://community.hortonworks.com/questions/69004/nifi-monitoring-processor-and-nifi-service.html