Variable registry update is behaving stuck at 'Stopping affected Processors' though all effected processors are in STOPPED state

[#1]

Hello Experts,

I am using below end point to send update variable registry request -

"/process-groups/<id>/variable-registry/update-requests" and check the status using below url in loop "/process-groups/<id>/variable-registry/update-requests/<variableRegistryUpdateId>".

It works well for some time and all of a sudden it starts behaving strange.

Status will not get change even after 200 times status api call with almost 2 mins wait total.
If we see the status api response , it shows like below - where it is stuck at 'Stopping affected Processors' stage , but if we look at effected processors, in one case there is not effected processors to stop and in another case all processors are already in stopped state but still it is not at progressing.

Any idea what could be the reason?

Update status check response 

Case 1 :

{
"request": {
"requestId": "573b0dd9-6f14-4ca2-80d1-513d93451f22",
"uri": "https://o9nifi-1:9091/nifi-api/process-groups/9e391108-646a-3c3c-a642-a4fbce1d9107/variable-registry...",
"submissionTime": "04/24/2025 13:47:09.647 GMT",
"lastUpdated": "04/24/2025 13:47:09.647 GMT",
"complete": false,
"percentCompleted": 0,
"updateSteps": [
{
"description": "Identifying components affected",
"complete": true
},
{
"description": "Stopping affected Processors",
"complete": false
},
{
"description": "Disabling affected Controller Services",
"complete": false
},
{
"description": "Applying Updates",
"complete": false
},
{
"description": "Re-Enabling affected Controller Services",
"complete": false
},
{
"description": "Restarting affected Processors",
"complete": false
}
],
"processGroupId": "9e391108-646a-3c3c-a642-a4fbce1d9107",
"affectedComponents": []
}
}

Case 2:

{
"request" : {
"requestId" : "b5ed4d79-13d2-4dfb-b556-8409e1ddfea6",
"uri" : "https://o9nifi-1:9091/nifi-api/process-groups/9e391108-646a-3c3c-a642-a4fbce1d9107/variable-registry...",
"submissionTime" : "04/24/2025 16:52:00.067 GMT",
"lastUpdated" : "04/24/2025 16:52:00.067 GMT",
"complete" : false,
"percentCompleted" : 0,
"updateSteps" : [ {
"description" : "Identifying components affected",
"complete" : true
}, {
"description" : "Stopping affected Processors",
"complete" : false
}, {
"description" : "Disabling affected Controller Services",
"complete" : false
}, {
"description" : "Applying Updates",
"complete" : false
}, {
"description" : "Re-Enabling affected Controller Services",
"complete" : false
}, {
"description" : "Restarting affected Processors",
"complete" : false
} ],
"processGroupId" : "9e391108-646a-3c3c-a642-a4fbce1d9107",
"affectedComponents" : [ {
"revision" : {
"clientId" : "685687f8-0196-1000-340d-d586d2c6d581",
"version" : 10
},
"id" : "2b0e0d15-5e1c-341c-87b1-421a39a81200",
"permissions" : {
"canRead" : true,
"canWrite" : true
},
"bulletins" : [ ],
"component" : {
"processGroupId" : "8a5d122b-b08b-30aa-b794-a2884fd31d45",
"id" : "2b0e0d15-5e1c-341c-87b1-421a39a81200",
"referenceType" : "PROCESSOR",
"name" : "O9QueueRouting",
"state" : "STOPPED",
"activeThreadCount" : 0
},
"processGroup" : {
"id" : "8a5d122b-b08b-30aa-b794-a2884fd31d45",
"name" : "test1"
},
"referenceType" : "PROCESSOR"
}, {
"revision" : {
"clientId" : "685687f8-0196-1000-340d-d586d2c6d581",
"version" : 5
},
"id" : "11479904-0add-3d9f-b489-1024c46275c3",
"permissions" : {
"canRead" : true,
"canWrite" : true
},
"bulletins" : [ ],
"component" : {
"processGroupId" : "7a9a49dc-b361-394b-93d2-6cfb83f6b85c",
"id" : "11479904-0add-3d9f-b489-1024c46275c3",
"referenceType" : "PROCESSOR",
"name" : "RouteOnAttribute",
"state" : "STOPPED",
"activeThreadCount" : 0
},
"processGroup" : {
"id" : "7a9a49dc-b361-394b-93d2-6cfb83f6b85c",
"name" : "CheckRequestStatus"
},
"referenceType" : "PROCESSOR"
}, {
"revision" : {
"version" : 0
},
"id" : "2b0e0d15-5e1c-341c-a8d0-58edec551d2d",
"permissions" : {
"canRead" : true,
"canWrite" : true
},
"bulletins" : [ ],
"component" : {
"processGroupId" : "8a5d122b-b08b-30aa-a844-848d88e69396",
"id" : "2b0e0d15-5e1c-341c-a8d0-58edec551d2d",
"referenceType" : "PROCESSOR",
"name" : "O9QueueRouting",
"state" : "STOPPED",
"activeThreadCount" : 0
},
"processGroup" : {
"id" : "8a5d122b-b08b-30aa-a844-848d88e69396",
"name" : "two1"
},
"referenceType" : "PROCESSOR"
}, {
"revision" : {
"version" : 0
},
"id" : "2b0e0d15-5e1c-341c-893a-07179a37b42c",
"permissions" : {
"canRead" : true,
"canWrite" : true
},
"bulletins" : [ ],
"component" : {
"processGroupId" : "8a5d122b-b08b-30aa-a4a8-c6402262c60e",
"id" : "2b0e0d15-5e1c-341c-893a-07179a37b42c",
"referenceType" : "PROCESSOR",
"name" : "O9QueueRouting",
"state" : "STOPPED",
"activeThreadCount" : 0
},
"processGroup" : {
"id" : "8a5d122b-b08b-30aa-a4a8-c6402262c60e",
"name" : "two22"
},
"referenceType" : "PROCESSOR"
}, {
"revision" : {
"version" : 0
},
"id" : "2b0e0d15-5e1c-341c-acab-14e6adfe64da",
"permissions" : {
"canRead" : true,
"canWrite" : true
},
"bulletins" : [ ],
"component" : {
"processGroupId" : "8a5d122b-b08b-30aa-b227-72500a7c62cc",
"id" : "2b0e0d15-5e1c-341c-acab-14e6adfe64da",
"referenceType" : "PROCESSOR",
"name" : "O9QueueRouting",
"state" : "STOPPED",
"activeThreadCount" : 0
},
"processGroup" : {
"id" : "8a5d122b-b08b-30aa-b227-72500a7c62cc",
"name" : "two33"
},
"referenceType" : "PROCESSOR"
}, {
"revision" : {
"clientId" : "685687f8-0196-1000-340d-d586d2c6d581",
"version" : 5
},
"id" : "2c48b295-2093-3bd2-a45a-d50c6d95cb8b",
"permissions" : {
"canRead" : true,
"canWrite" : true
},
"bulletins" : [ ],
"component" : {
"processGroupId" : "7a9a49dc-b361-394b-93d2-6cfb83f6b85c",
"id" : "2c48b295-2093-3bd2-a45a-d50c6d95cb8b",
"referenceType" : "PROCESSOR",
"name" : "ReplaceText",
"state" : "STOPPED",
"activeThreadCount" : 0
},
"processGroup" : {
"id" : "7a9a49dc-b361-394b-93d2-6cfb83f6b85c",
"name" : "CheckRequestStatus"
},
"referenceType" : "PROCESSOR"
} ]
}
}

Thanks,

Mahendra

[#2]

@hegdemahendra 

My initial guess would be a long running or hung thread on processor preventing it from transitioning from "stopping" to "stopped" state required.  Including your full Apache NiFi version is aways helpful when posting a question.  Allows someone helping to check easier if there happens to be any known bugs in that release that may be related to the issue.

Analyzing NiFi thread dumps might help narrowing down what is specifically being waited on.

NOTE: The Apache NiFi 1.x version are going end of life and Apache NIFi 2.x no longer supports variable registry.  NiFi Parameters also available in the newer Apache NiFi 1.x versions are the replacement.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3]

@MattWho  thank you so much.
Apologies missed to mention version.
I am using Apache Nifi 1.25.0 and running nifi in k8 with 2 node or 1 node cluster.

Variable registry update will be trying to stop only effected processors right?
In one case there is no effected processors for the variable I am trying to use.

Also can you guide on collecting thread dump?

./nifi.sh dump <filename fo dump>

Is this the command I should be using?

Thank you again for you help.

[#4]

@hegdemahendra 

In Apache NiFi 1.x versions the nifi.sh dump capability still exists, so you can still use that.

./nifi.sh dump /tmp/<dump filename>

NOTE: dump is no longer an option in Apache NiFi 2.x versions

But since NiFi is a Java application you can also take a java thread dump:

jmap -dump:format=b,file=heap.hprof <nifi pid>

Keep in mind that NiFi will have two process running (bootstrap process and main process). Make sure you are using the main NiFi pid and not the pid for the bootstrap process.

Newer version of 1.x also include a diagnostics option.   When in verbose, it will also output the thread dump in the diagnostics output.

./nifi.sh diagnostics --verbose /tmp/diag.txt

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#5]

@MattWho 
I see below logs in dumps, apart from these I dont see anything like waiting etc

"Variable Registry Update Thread" Id=70591 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f1aa08e
at java.base@11.0.22/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11.0.22/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.base@11.0.22/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.base@11.0.22/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)
at java.base@11.0.22/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.base@11.0.22/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base@11.0.22/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base@11.0.22/java.lang.Thread.run(Unknown Source)

-------------
"Timer-Driven Process Thread-102" Id=1030 BLOCKED on org.apache.nifi.controller.scheduling.LifecycleState@3ed4243f
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:147)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.base@11.0.22/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base@11.0.22/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.base@11.0.22/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base@11.0.22/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base@11.0.22/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base@11.0.22/java.lang.Thread.run(Unknown Source)
Number of Locked Synchronizers: 1
- java.util.concurrent.ThreadPoolExecutor$Worker@2843111c

Thanks

mahendra