@hegdemahendra  Did you take heap dumps to confirm which class was consuming the heap? Any thread dump analysis when heap usage was growing? What incorrect values were configured that you feel led to this component consuming large amounts of heap memory? Not really finding any known issues of memory with this consumeKafka processor.  Any details you can provide (processor configuration, log exceptions while it was running with bad config, etc) may help. Thank you, Matt ... View more

@Shelton Thank you for the detailed answer, much appreciated ! ... View more

@MattWho  I see below logs in dumps, apart from these I dont see anything like waiting etc   "Variable Registry Update Thread" Id=70591 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f1aa08e at java.base@11.0.22/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.22/java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.base@11.0.22/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.base@11.0.22/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source) at java.base@11.0.22/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.base@11.0.22/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base@11.0.22/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base@11.0.22/java.lang.Thread.run(Unknown Source)   ------------- "Timer-Driven Process Thread-102" Id=1030 BLOCKED on org.apache.nifi.controller.scheduling.LifecycleState@3ed4243f at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:147) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base@11.0.22/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base@11.0.22/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base@11.0.22/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base@11.0.22/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base@11.0.22/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base@11.0.22/java.lang.Thread.run(Unknown Source) Number of Locked Synchronizers: 1 - java.util.concurrent.ThreadPoolExecutor$Worker@2843111c Thanks mahendra ... View more

Thank you so much @MattWho for the detailed answer. The retry logic helped a lot, I have added 'RetryFlowFile' processors in between to avoid infinite loop of retry. ... View more

@hegdemahendra  As far as your issue goes, it would probably be useful to collect a series of thread dumps (at least spaced 5 minutes apart).  Then you would be looking for any threads related to the stopping of components to see if they are progressing or hung.  Is it stuck on stopping a specific processor or processor class?  Do any of the processors that are being stopped have active threads showing for them? Thank you, Matt ... View more

Hi @hegdemahendra I am also facing same issue  With FetchDistributedMapCache. How can i check the size of the flowfile we are reading, for me also it is set to 1Mb but the data we are saving in cache should not be 1Mb as we are barely saving 100characters in the flowfile payload. How can we check the data size we are trying to get from cache? Thanks Akash ... View more

@hegdemahendra  The FlowFile connection back pressure thresholds are soft limits.  Once one of the configured back pressure thresholds on reached or exceeded, NiFi will not allow the processor feeding that connection to get scheduled to execute again. So in your case no back pressure is being applied, so the ConsumeAzureEventHub processor is being allowed to scheduled to execute.  During a single execution it is consuming more events then the threshold settings.   What is the batch size set to in your ConsumeAzureEventHub processor? Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@araujo @bbende @MattWho - do you have any suggestions? ... View more

Hello @Mais - Were you able to deserialise and consume both key & value ? In my case I am able to get deserialised value but dont see key anywehere! ... View more

What an explanation ! Cleared my doubts. Thank you so much @MattWho .  ... View more