We have 3 node apache nifi cluster, built on same versions of nifi i.e Apache nifi 1-.16.0 and Open JDK 11. When trying to delete a funnel from below loop, one of the node gives an error on UI "###### node is is unable to fulfill this request due to: An unexpected error has occurred. Please check the logs for additional details."
Enabled debug logs to see more details and unfortunately it ended up in java.lang.StackOverflow error from the below logs. Seems like flow ended in weird state. We tried deleting work directories on cluster and restarting cluster. It din't help though
ERROR [NiFi Web Server-34943] o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: java.lang.StackOverflowError: Delayed StackOverflowError due to ReservedStackAccess annotated method. R eturning Internal Server Error response.
91796 java.lang.StackOverflowError: Delayed StackOverflowError due to ReservedStackAccess annotated method
91797 at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1382)
91798 at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:897)
91799 at org.apache.nifi.controller.StandardFunnel.getIncomingConnections(StandardFunnel.java:268)
@Chaitanya_,
First of all, why the infinite loop? What is the point of that and what where you actually trying to achieve? ( just out of curiosity)
Now, in terms of your problem, I can give you two possible scenarios which you could try, which might eventually help you:
1) Add a new processor on your Canvas (LogMessage for example) and try moving the queue from the center funnel towards the right funnel into the new processor. Please make sure that all your processors are stopped and disabled and your queues are empty. This should allow you to move the queue without any issues, while avoiding the infinite loop, which will eventually help you to remove the funnels from your canvas. Another thing, try also removing the purple highlighted queue, so you are certain that the loop is no longer a loop. Afterwards, you should be able to remove all the queues, starting from right to left.
2) This might be a little hard and requires plenty of attention (and it is not really recommended), but in times of desperation, you could try manually modifying the flow.xml.gz and the flow.json.gz and remove the parts of those funnels. You then can upload the new version of files in all NiFi nodes and you should no longer see those funnels on your canvas. However, before doing this, make sure that you create a backup of those files, in case you mess something up. Nevertheless, this is not really recommended so I highly advise you to try with the first solution before even trying this one.
PS: make sure that all your nodes are up and running. Or stop the nodes, work on a single node and copy the flow.xml and flow.json to all the other nodes and start them.
Hope it helps!
@Chaitanya_,
First of all, why the infinite loop? What is the point of that and what where you actually trying to achieve? ( just out of curiosity)
Now, in terms of your problem, I can give you two possible scenarios which you could try, which might eventually help you:
1) Add a new processor on your Canvas (LogMessage for example) and try moving the queue from the center funnel towards the right funnel into the new processor. Please make sure that all your processors are stopped and disabled and your queues are empty. This should allow you to move the queue without any issues, while avoiding the infinite loop, which will eventually help you to remove the funnels from your canvas. Another thing, try also removing the purple highlighted queue, so you are certain that the loop is no longer a loop. Afterwards, you should be able to remove all the queues, starting from right to left.
2) This might be a little hard and requires plenty of attention (and it is not really recommended), but in times of desperation, you could try manually modifying the flow.xml.gz and the flow.json.gz and remove the parts of those funnels. You then can upload the new version of files in all NiFi nodes and you should no longer see those funnels on your canvas. However, before doing this, make sure that you create a backup of those files, in case you mess something up. Nevertheless, this is not really recommended so I highly advise you to try with the first solution before even trying this one.
PS: make sure that all your nodes are up and running. Or stop the nodes, work on a single node and copy the flow.xml and flow.json to all the other nodes and start them.
Hope it helps!
