@Shelton Thank you for the detailed answer, much appreciated ! ... View more

@MattWho  I see below logs in dumps, apart from these I dont see anything like waiting etc   "Variable Registry Update Thread" Id=70591 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f1aa08e at java.base@11.0.22/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.22/java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.base@11.0.22/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.base@11.0.22/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source) at java.base@11.0.22/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.base@11.0.22/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base@11.0.22/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base@11.0.22/java.lang.Thread.run(Unknown Source)   ------------- "Timer-Driven Process Thread-102" Id=1030 BLOCKED on org.apache.nifi.controller.scheduling.LifecycleState@3ed4243f at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:147) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base@11.0.22/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base@11.0.22/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base@11.0.22/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base@11.0.22/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base@11.0.22/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base@11.0.22/java.lang.Thread.run(Unknown Source) Number of Locked Synchronizers: 1 - java.util.concurrent.ThreadPoolExecutor$Worker@2843111c Thanks mahendra ... View more

Thank you so much @MattWho for the detailed answer. The retry logic helped a lot, I have added 'RetryFlowFile' processors in between to avoid infinite loop of retry. ... View more

@hegdemahendra  As far as your issue goes, it would probably be useful to collect a series of thread dumps (at least spaced 5 minutes apart).  Then you would be looking for any threads related to the stopping of components to see if they are progressing or hung.  Is it stuck on stopping a specific processor or processor class?  Do any of the processors that are being stopped have active threads showing for them? Thank you, Matt ... View more

Hi @hegdemahendra I am also facing same issue  With FetchDistributedMapCache. How can i check the size of the flowfile we are reading, for me also it is set to 1Mb but the data we are saving in cache should not be 1Mb as we are barely saving 100characters in the flowfile payload. How can we check the data size we are trying to get from cache? Thanks Akash ... View more

@hegdemahendra  The FlowFile connection back pressure thresholds are soft limits.  Once one of the configured back pressure thresholds on reached or exceeded, NiFi will not allow the processor feeding that connection to get scheduled to execute again. So in your case no back pressure is being applied, so the ConsumeAzureEventHub processor is being allowed to scheduled to execute.  During a single execution it is consuming more events then the threshold settings.   What is the batch size set to in your ConsumeAzureEventHub processor? Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@araujo @bbende @MattWho - do you have any suggestions? ... View more

Hello @Mais - Were you able to deserialise and consume both key & value ? In my case I am able to get deserialised value but dont see key anywehere! ... View more

What an explanation ! Cleared my doubts. Thank you so much @MattWho .  ... View more

Hello @hegdemahendra  Always very helpful if you include the exact version of Apache NiFI, Cloudera HDF, or Cloudera CFM being used. My guess here would be one or both of the following: You have multiple FlowFiles all pointing at the same content claims queued in connections within your dataflow(s) on the canvas.  As long as a FlowFile exists on the canvas it will exist in flowfile_repository.   Users should avoid leaving FlowFiles queued in connection on NiFi. Some users tend to allow FlowFile to accumulate at stopped processor components rather then auto-terminate them.  Even if a FlowFile does not have any content its FlowFile attributes/metadata still consume disk space. You are extracting content from your FlowFiles into FlowFile attributes resulting in large FlowFile attribute/metadata being stored in the flowfile_repository.   Dataflow designers should avoid extracting large amounts flowfile content in to the FlowFile's attributes.  Instead try to build dataflows and utilize components that read content from the FlowFile's content instead of from FlowFile attributes. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more