Can we capture the run details of processor and process group?

[#1]

Hi All,

Let's say I have a Process Group that runs once per day and contains a set of processors. What I would like to track is:

When the Process Group started

How long it ran

When it completed

...both at the Process Group level and the individual processor level within the group.

Can we capture this information from NiFi logs? If these details are not available in the logs, where else can I find them?

Basically, I'm working on building a centralized table to store daily run details for each Process Group.

[#2]

@justloseit Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @SAMSAL @joseomjr  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3]

@justloseit 

NiFi Process groups are just logical containers for processors.  A Process group does not run/execute.  Selecting "Start" on a process group triggers starting of all the components within that process group. 

In your case it sounds like you have have setup cron scheduling on your ingest/starting processor(s) within the process group.  All downstream processors to that source should be set run all the time and not cron based scheduling.  So what you are really looking for is how long it took the processors within that process group to process all produced FlowFiles to point of termination?

Besides looking at the lineage data for each FlowFile that traverses all the processor in a process group, I can't think of how else you would get that data.   Take a look at the SiteToSiteProvenanceReportingTask available in Apache NiFi.  It allows you send the provenance data (produces a lot of data depending on size of yoru dataflows and amount of FlowFiles being processed) via NiFi's Site-To-Site protocol to another NiFi instance (would recommend a separate dedicated NiFi to receive this data).  You can then build a  dataflow to process that data how you want to retain what information you need, or send it to an external storage/processing system.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt