Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Not able to delete an inifinite loop built with funnels in nifi

avatar
New Contributor

We have 3 node apache nifi cluster, built on same versions of nifi i.e Apache nifi 1-.16.0 and Open JDK 11. When trying to delete a funnel from below loop, one of the node gives an error on UI "###### node is is unable to fulfill this request due to: An unexpected error has occurred. Please check the logs for additional details."


Nifi infinite loopNifi infinite loop

Enabled debug logs to see more details  and unfortunately it ended up in  java.lang.StackOverflow error from the below logs. Seems like flow ended in weird state.  We tried deleting work directories on cluster and restarting cluster. It din't help though 

ERROR [NiFi Web Server-34943] o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: java.lang.StackOverflowError: Delayed StackOverflowError due to ReservedStackAccess annotated method. R eturning Internal Server Error response.
91796 java.lang.StackOverflowError: Delayed StackOverflowError due to ReservedStackAccess annotated method
91797 at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1382)
91798 at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:897)
91799 at org.apache.nifi.controller.StandardFunnel.getIncomingConnections(StandardFunnel.java:268)

 

1 ACCEPTED SOLUTION

avatar

@Chaitanya_,

First of all, why the infinite loop? What is the point of that and what where you actually trying to achieve? ( just out of curiosity)

Now, in terms of your problem, I can give you two possible scenarios which you could try, which might eventually help you:

1) Add a new processor on your Canvas (LogMessage for example) and try moving the queue from the center funnel towards the right funnel into the new processor. Please make sure that all your processors are stopped and disabled and your queues are empty. This should allow you to move the queue without any issues, while avoiding the infinite loop, which will eventually help you to remove the funnels from your canvas. Another thing, try also removing the purple highlighted queue, so you are certain that the loop is no longer a loop. Afterwards, you should be able to remove all the queues, starting from right to left.

cotopaul_1-1699260145186.png

 

2) This might be a little hard and requires plenty of attention (and it is not really recommended), but in times of desperation, you could try manually modifying the flow.xml.gz and the flow.json.gz and remove the parts of those funnels. You then can upload the new version of files in all NiFi nodes and you should no longer see those funnels on your canvas. However, before doing this, make sure that you create a backup of those files, in case you mess something up. Nevertheless, this is not really recommended so I highly advise you to try with the first solution before even trying this one.

PS: make sure that all your nodes are up and running. Or stop the nodes, work on a single node and copy the flow.xml and flow.json to all the other nodes and start them.

Hope it helps!

View solution in original post

4 REPLIES 4

avatar
Community Manager

@Chaitanya_ Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @steven-matison @MattWho  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.


Regards,

Diana Torres,
Community Moderator


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:

avatar
New Contributor

Thanks @DianaTorres for tagging Nifi experts!
I thought of providing more inputs would help. 

When we came to know about the issue, one of the cluster nodes was disconnected from cluster. After a restart on cluster and emptying data in queue, all cluster operations are  normal. I was able to add and delete processors with out any issue. When we are trying to delete a funnel from infinite loop, it is that specific node which was down not able to fullfill the request. 

I even tried deleting flow.xml.gz & flow.json.gz from the node, but it couldn't solve the issue.

avatar

@Chaitanya_,

First of all, why the infinite loop? What is the point of that and what where you actually trying to achieve? ( just out of curiosity)

Now, in terms of your problem, I can give you two possible scenarios which you could try, which might eventually help you:

1) Add a new processor on your Canvas (LogMessage for example) and try moving the queue from the center funnel towards the right funnel into the new processor. Please make sure that all your processors are stopped and disabled and your queues are empty. This should allow you to move the queue without any issues, while avoiding the infinite loop, which will eventually help you to remove the funnels from your canvas. Another thing, try also removing the purple highlighted queue, so you are certain that the loop is no longer a loop. Afterwards, you should be able to remove all the queues, starting from right to left.

cotopaul_1-1699260145186.png

 

2) This might be a little hard and requires plenty of attention (and it is not really recommended), but in times of desperation, you could try manually modifying the flow.xml.gz and the flow.json.gz and remove the parts of those funnels. You then can upload the new version of files in all NiFi nodes and you should no longer see those funnels on your canvas. However, before doing this, make sure that you create a backup of those files, in case you mess something up. Nevertheless, this is not really recommended so I highly advise you to try with the first solution before even trying this one.

PS: make sure that all your nodes are up and running. Or stop the nodes, work on a single node and copy the flow.xml and flow.json to all the other nodes and start them.

Hope it helps!

avatar
New Contributor

Thanks a lot! @cotopaul 

Solution 1 worked in our case. We stopped our processors and emptied queues, after that moving the connection from center funnel to the new processor helped us to remove other funnels.
To answer your question, why the infinite loop ? it was effect of blunder - when I was working with the funnels weren't located so sparsely, so I didn't see that the connection I'm going to make and created a loop