NiFi: Provenance list capture

ashsskum — Tue, 06 Dec 2016 13:07:14 GMT

Hi,

I wanted to be able to track how many message (files) were consumed by the ConsumeKafka process and size of it, for a particular time window. With reference to that, please advise on the below:

1) Can we be able to find the list of items (see attached, such as file size, queued duration) that got processed, even when they are no longer in the queue?

2) Can we query items in queue specific to a particular processor?

3) Is it possible to retrieve and post the above to a custom Amabari metrics?

The objective is to maintain a trend of the average message size and the number of messages, specific to a particular topic (configured in the ConsumeKafka processor).

Re: NiFi: Provenance list capture

gkeys — Tue, 06 Dec 2016 19:52:53 GMT

There is both a push (reporting api) and pull (restful api) way to automate metrics collection in NiFI. See this post for an overview: https://community.hortonworks.com/questions/69004/nifi-monitoring-processor-and-nifi-service.html

question Re: NiFi: Provenance list capture in Archives of Support Questions (Read Only)

NiFi: Provenance list capture

Re: NiFi: Provenance list capture