RouteOnAttribute Processor will not Process data

[#1]

I have a nifi instance with numerous processors that are functioning fine.

Except one RouteOnAttribute Processor that has a 90GB queue before it. The processor is on, and it has possible thread count of 10. It is not currently running any threads.

The nifi instance max thread count is 100 (event and timer driven), and the current number of threads running is 1, for a getsftp processor that is running.

Heap usage is 32% and the content and flowfile repos are at 2%.

The only thing that processes this queue is if I copy the route on attribute processor (and its attached putsftp processors) and move the queue over to the new copied RouteOnAttribute Processor and hit play.

Any ideas on this?

[#2]

When its stuck can you run "bin/nifi.sh dump" and get the entire thread dump from nifi-bootstrap.log and attach it?

Also, what version of NiFi?

[#3]

I am using Apache nifi 1.1.1.

I did this but the problem is It's on a network that I can only send data into. I can't take any data out.

I see a bunch of Event Driven Process thread-92 id=89206 Time_waiting

Anything in particular we should be looking for?

John

[#4]

Basically just looking for a thread that could be stuck and preventing something from happening. Unfortunately there isn't something specific I could say to look for, maybe a stack that looks very different than all the other threads, or one that always seems to be in the exact same spot every time you run dump.

You could look for RouteOnAttribute in the dump, but since you said the processor itself doesn't show any threads running then I doubt that will show up, but worth a try.

[#5]

Thanks for your help on this! It turns out there was back-pressure issue on later processors. The new default of NiFi 1.0+ is 1GB. I was just made aware of this....

Check out my new question on how to change the default value for back-pressure.

https://community.hortonworks.com/questions/97788/change-default-connection-back-pressure-threshold....

[#6 · Accepted solution]

@John T

What version of NiFi are you running?

Do any of the connections leading from the RouteOnAttribute processor have back pressure being applied? If you are running a new enough version the connection will be highlighted red when back pressure is being applied. If back pressure is being applied the source processor to that connection will not be allowed to run.

Can you share a screenshot of your dataflow?

If the above is not the case, have you checked your nifi-app.log for any errors? look for Out of Memory (OOM) ERRor for example.

Thanks,

Matt

[#7]

@Matt Clarke

Thanks for the quick response! we are using 1.1.1.

None of the processors before or after have any back-pressure applied.

I also checked my nifi-app.log and did not see any OOMs, I checked our heap and it was at 30 some percent when I checked. The whole nifi-app.log actually outputs nothing when this happens.

As I told Bryan, this is on a network that can only receive data, no data can come out, so getting a screenshot would be hard, but its basically 1 input queue to a route on attribute proccessor that is routing on 1 attrbiute and has about 40 possible routes.

John

[#8]

Is NiFi reporting a lot of Garbage collection going on?

Do you have a large number of FlowFiles queued in your dataflow(s)?

Does a data provenance query run against your RouteOnAttribute processor show extremely slow processing of FlowFiles or a complete stoppage? Does a restart of NiFi trigger it to process files again?

[#9]

Thanks for your help on this! It turns out there was back-pressure issue on later processors. The new default of NiFi 1.0+ is 1GB. I was just made aware of this....

Check out my new question on how to change the default value for back-pressure.

https://community.hortonworks.com/questions/97788/change-default-connection-back-pressure-threshold....

[#10]

@John T

Sounded a lot like a back pressure scenario to me when you first described what was going on. Glad you were able to resolve you issue. I also saw your other post and commented on it.