Thank you for the detailed explanation, Matt ... View more

@MattWho  Thank you for your comprehensive response. We are using NIFI cluster topology and only two processors ConsumeKafka and PublishKafka. Our topics are configured with 36 partitions ,we have 12 nodes in our NIFI cluster, so we configured the concurrent tasks to 3 . As mentioned, our topics are defined with cleanup police : compact, which requires a Kakfa key for each record. Can we use ConsumeKafkaRecord and PublishKafkaRecord when a Kafka key is involved?  ... View more

Thank you for your insights @steven-matison ,I will check this out. ... View more

It looks like the high heap on the primary was caused by too many open files,Is there a way to identify the processor that open many files ? We are using openjdk 17.0.7 ,nifi v 1.21.0 and custom processors . @MattWho wrote: @edim2525  GC kicks in around 80% of heap memory usage.  You could certainly enable GC debug logging to verify that GC is executing.  GC can only clean-up unused memory (Memory no linger being held by a process).  I see you have three NiFi nodes. Are you only having Heap memory usage issues on the one node?   I see the node with growing heap usage is the elected primary node. What processors do you have running as "primary node" execution? Does your primary node have a lot more queued FlowFiles than the other nodes? If you disconnect the primary node from your cluster which will force a new primary node to be elected, does the heap then start to grow on the new elected primary node? What version of NiFi are you running? What version of Java is your NiFi running with? Have you collected heap dumps and analyzed them to see where the heap usage is being used? Do you have any custom processors added to your NiFi? Are you using any scripting based processors where you have written your own code that is being executed within NiFi? Matt 2023-11-11 04:13:18,240 ERROR [Timer-Driven Process Thread-30] o.a.n.p.kafka.pubsub.PublishKafka_2_6 PublishKafka_2_6[id=c8d6883a-8b07-3d38-b28f-fc8263c83a39] Processing halted: yielding [1 sec] java.lang.IllegalStateException: Cannot complete publishing to Kafka because Publisher Lease was already closed         at org.apache.nifi.processors.kafka.pubsub.PublisherLease.complete(PublisherLease.java:476)         at org.apache.nifi.processors.kafka.pubsub.PublishKafka_2_6.onTrigger(PublishKafka_2_6.java:490)         at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)         at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1360)         at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:246)         at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)         at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)         at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)         at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)         at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)         at java.base/java.lang.Thread.run(Thread.java:833) 2023-11-11 04:13:18,241 WARN [Timer-Driven Process Thread-30] o.a.n.controller.tasks.ConnectableTask Processing halted: uncaught exception in Component [PublishKafka_2_6[id=c8d6883a-8b07-3d38-b28f-fc8263c83a39]] java.lang.IllegalStateException: Cannot complete publishing to Kafka because Publisher Lease was already closed         at org.apache.nifi.processors.kafka.pubsub.PublisherLease.complete(PublisherLease.java:476)         at org.apache.nifi.processors.kafka.pubsub.PublishKafka_2_6.onTrigger(PublishKafka_2_6.java:490)         at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)         at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1360)         at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:246)         at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)         at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)         at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)         at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)         at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)         at java.base/java.lang.Thread.run(Thread.java:833) 2023-11-11 04:13:18,247 ERROR [Listen to Bootstrap] org.apache.nifi.BootstrapListener Failed to process request from Bootstrap due to java.io.IOException: Too many open files java.io.IOException: Too many open files         at java.base/sun.nio.ch.Net.accept(Native Method)         at java.base/sun.nio.ch.NioSocketImpl.timedAccept(NioSocketImpl.java:711)         at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:752)         at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:675)         at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:641) ... View more

Thank you for your reply, I'll check and update. ... View more