How to Track and Display Integration Execution Details in NiFi?

[#1]

Hi everyone,

I’m working on tracking and monitoring integration executions in Apache NiFi . I want to achieve the following:

1. Display all the configured integrations available to users so they can check their execution status.
2. Show all executions of a specific integration with details such as:
   • Start time
   • End time
   • Duration
   • Status (success/failure)
   • Total number of sub-items processed, including success and failure counts.
3. Track all integrations that happened as part of a single execution and display:
   • Start time
   • End time
   • Duration
   • Status
   • Any relevant messages or error details

I’d like to know the best way to implement this in NiFi. Are there recommended processors, templates, or logging strategies that can help achieve this? Should I be using provenance data, custom logging, or NiFi's built-in monitoring tools?

Any insights, examples, or best practices would be greatly appreciated!

Thanks in advance!

[#2]

@ajay32 

Welcome to the Community!

Unfortunately, I don't think you have provided enough detail to answer your questions.

NiFi is typically is used to build dataflows that are always running, so a clear well defined use case for you dataflow may help the community to provide you with responses.

1. "Display all the configured integrations available to users" <-- not clear on yoru ask here.  What do you mean by "configured integrations"?  Is an integration an End-to-End NiFi Dataflow build on the NiFi canvas?
2. Show execution details:
   1. Start Time - Assuming NiFi dataflow, how are your processor components being scheduled?  (Cron driven?, Timer Driven?)
      
   2. End Time - Downstream processors are not typically scheduled using cron, but rather always running using timer driven to ensure they can process NiFi Flow Files as they are received.  There is no communication between one processor and another.  NiFi processors simply execute against a NiFi FlowFile and that FlowFile is passed to the next downstream processor.  That downstream processor is going to be unaware of how many FlowFiles to expect from the upstream processor. 
      
   3. Duration - The execution of your first Processor in a dataflow will produce a NiFi FlowFile with a timestamp.  You could compare that timestamp with current time at end of your complete dataflow execution to calculate how long it took for that FlowFile to process through all NiFi processor components.
      
   4. Status - What constitutes a "success" and "failure" in your dataflow / use case?
   5. Sub-items - What is a "sub-item" in your dataflow?
      
3. Track all integrations:
   1. You can build each unique dataflow in a NiFi process group and enable per process group logging by configuring a unique "Log File Suffix" within the process group configuration.
   2. NiFi provenance is also an option.  If you are dealing with multiple unique dataflows, modifying processor names so that those in one complete dataflow share the prefix ("flow1_<processor name>") would make it easier to correlate all events related to a single datflow.  There are a few Provenance NiFi reporting tasks that can be used.

Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3]

@ajay32 Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.