question Re: In NiFi queue with Load Balance Strategy RoundRobin flowfiles stuck in Support Questions

In NiFi queue with Load Balance Strategy RoundRobin flowfiles stuck

ilyal — Thu, 17 Oct 2019 13:35:15 GMT

Hi All

If in connection(queue between two processors) I switch on RoundRobin strategy flow files start to stuck there for minutes.

Some investigation showed, that flow files stuck only on the primary node.

No any Warns or Errors in logs.

I am ready to provide any information

Please, any help.

Thanks

Re: In NiFi queue with Load Balance Strategy RoundRobin flowfiles stuck

MattWho — Thu, 17 Oct 2019 13:54:19 GMT

@ilyal

What version of NiFi are you running? Is it Apache NiFi 1.8.x?

There are numerous bugs with the new load balanced connections feature. The good number of these known bugs have been addressed between NiFi 1.9.0 and NiFi 1.9.2.

https://issues.apache.org/jira/browse/NIFI-5745

https://issues.apache.org/jira/browse/NIFI-5919

https://issues.apache.org/jira/browse/NIFI-5663

https://issues.apache.org/jira/browse/NIFI-5771

https://issues.apache.org/jira/browse/NIFI-6017

There are still some additional bugs that are fixed in NiFi 1.10.0

https://issues.apache.org/jira/browse/NIFI-6353

https://issues.apache.org/jira/browse/NIFI-6760

https://issues.apache.org/jira/browse/NIFI-6517

https://issues.apache.org/jira/browse/NIFI-6736

https://issues.apache.org/jira/browse/NIFI-6285

https://issues.apache.org/jira/browse/NIFI-6759

I strongly recommend upgrading to Apache NiFi 1.10 as a first step upgrading (releasing soon).

Hope this helps,
Matt

Re: In NiFi queue with Load Balance Strategy RoundRobin flowfiles stuck

ilyal — Sun, 27 Oct 2019 07:27:04 GMT

Thanks you Matt. I will try and will update Question

Re: In NiFi queue with Load Balance Strategy RoundRobin flowfiles stuck

venu413 — Mon, 06 Jan 2020 16:25:37 GMT

Upgraded to 1.10 and now I see flow files are stuck with Load Balance strategy Partition By Attribute

Re: In NiFi queue with Load Balance Strategy RoundRobin flowfiles stuck

venu413 — Mon, 06 Jan 2020 16:31:33 GMT

@MattWho Upgraded to Nifi 1.10 recently and flowfiles are struck with LoadBalance strategy PartitionBy Attribute

Re: In NiFi queue with Load Balance Strategy RoundRobin flowfiles stuck

ilyal — Tue, 07 Jan 2020 06:53:45 GMT

Hi, I have issues with other processors, but some processors do not have this issue for example UpdateAttributes. If you add this processor it will be work.

Re: In NiFi queue with Load Balance Strategy RoundRobin flowfiles stuck

venu413 — Tue, 07 Jan 2020 18:58:08 GMT

Issue seems to be in Queues rather than processor. I tried with different processors and issue remains same For eg: below is the queue that points to updateAttribute Processor.

Re: In NiFi queue with Load Balance Strategy RoundRobin flowfiles stuck

MattWho — Tue, 07 Jan 2020 22:08:28 GMT

@venu413

If you open the NiFi summary UI (NiFi UI --> Global menu --> Summary), select the connections tab, locate this connection with the 54 queued flowfiles, and then click the cluster connection summary icon ()to far right, Are all 54 queued FlowFiles on same node?

Is anything being logged in the nifi-app.log on that node were these FlowFiles are queued?

Any observed errors in nifi-app.log during startup if you restart this node?

In your nifi.properties file, what values are configured for these properties:
nifi.cluster.load.balance.comms.timeout=
nifi.cluster.load.balance.connections.per.node=
nifi.cluster.load.balance.host=
nifi.cluster.load.balance.max.thread.count=
nifi.cluster.load.balance.port=
nifi.cluster.node.address=

If recommend that both the "nifi.cluster.node.address=" and "nifi.cluster.load.balance.host=" have been configured uniquely per node in your cluster to the resolvable hostname for the given node. So if you node has a hostname of node1.mycompany.com, then this hostname should be used in both these properties in the NiFi running on that host. Restart is needed anytime you edit the nifi.properties file.

Re: In NiFi queue with Load Balance Strategy RoundRobin flowfiles stuck

JatinSab — Mon, 10 Feb 2020 16:51:28 GMT

@MattWho Here are the values of properties asked

nifi.cluster.load.balance.host=nifi-dev-0.nifi-dev
nifi.cluster.load.balance.port=6342
nifi.cluster.load.balance.connections.per.node=10
nifi.cluster.load.balance.max.thread.count=8
nifi.cluster.load.balance.comms.timeout=300 sec

nifi.cluster.node.address=nifi-dev-0.nifi-dev

Still the issue is appearing.
Actually we have a Kubernetes Cluster using NiFi 1.11.1 image. However, it still appears to happen

Re: In NiFi queue with Load Balance Strategy RoundRobin flowfiles stuck

MattWho — Tue, 11 Feb 2020 16:20:18 GMT

@JatinSab

Apache NiFi 1.11.1 specifically has a fix https://jira.apache.org/jira/browse/NIFI-7059 which introduced a bug with load-balanced connections. This bug is addressed in 1.11.2 and is covered in jira https://jira.apache.org/jira/browse/NIFI-7117.

Thanks,
Matt

Re: In NiFi queue with Load Balance Strategy RoundRobin flowfiles stuck

MattWho — Tue, 11 Feb 2020 16:27:09 GMT

@venu413

Are these the same FlowFiles that were stuck prior to the upgrade?
If you clean out your connections, do you see the issue occur later?

Re: In NiFi queue with Load Balance Strategy RoundRobin flowfiles stuck

JatinSab — Tue, 11 Feb 2020 16:33:35 GMT

@MattWho no these are not same flowfiles before upgrade..
We cleaned all flow files before upgrade...

Re: In NiFi queue with Load Balance Strategy RoundRobin flowfiles stuck

MattWho — Tue, 11 Feb 2020 18:34:10 GMT

@JatinSab

Troubleshooting this here with limited information is going to prove difficult.
Does the downstream processor of the connection with the "stuck" FlowFiles have...
1. Configured for all nodes or primary node Execution? If primary node only, are the queued FlowFiles on the primary node?
2. Does processor show it has active threads? (small number in upper right corner of processor) . If so, get some thread dumps from your NiFi and inspect if the thread is progressing or hung.
3. Is back pressure being applied on any of the outbound connections?
4. Is processor getting threads to do work? What is CPU load on your system and configured "Max Timer Driven Thread Count?" It may be set to low causing a thread starved situation for your dataflow.

Additionally, if you try to perform a "list queue" on the connection of issue, what result do you get?

If you have a support contract with Cloudera, you can open a support case and they can help you to a solution.

Thanks,

Matt

Re: In NiFi queue with Load Balance Strategy RoundRobin flowfiles stuck

JatinSab — Tue, 11 Feb 2020 18:40:14 GMT

Thanks @MattWho ..
Joe Asked me to build using PR# 4045, and test it...
M on it to verify if it fixed the issue...
Will let you know...

Thanks for your help..