Hi Community, I'm working on a NiFi setup where I use a dedicated template to track the status of FlowFiles from various other templates. The status of each FlowFile is logged in a specific pattern, and I'm using this pattern to extract and persist status information. Here's a brief overview of the current approach: TailFile Processor reads log entries from a specific log file. SplitText Processor splits the log content line by line. ExtractGrok Processor extracts relevant fields using a defined Grok pattern. ReplaceText Processor restructures the data to a desired format (e.g., JSON). PutDatabaseRecord Processor stores the structured data into a database. Problems Faced: Queue Build-Up & Performance Bottleneck: TailFile often brings in large chunks of data, especially under high log volume. The SplitText processor cannot keep up with the rate of incoming data. This leads to large unprocessed FlowFiles piling up in the queue. FlowFile Explosion & Choking: Once a large FlowFile is split, it results in a burst of many smaller FlowFiles. This sudden expansion causes congestion and chokes downstream processors. Repository Storage Issues: The above behavior leads to excessive usage of the FlowFile Repository, Content Repository, and Provenance Repository. Over time, this is causing storage concerns and performance degradation   My Question:  Is there a way to optimize this flow to: Reduce the memory and storage pressure on NiFi repositories? Handle incoming log data more efficiently without overwhelming the system? Or, is there a better architectural pattern to achieve log-based FlowFile tracking across templates? Any guidance or best practices would be greatly appreciated. Thanks! ... View more