question Re: Nifi: Flowfile stuck in front of a processor group when flowfile concurrency is set to single batch per node in Support Questions

Nifi: Flowfile stuck in front of a processor group when flowfile concurrency is set to single batch per node

yuanhao1999 — Tue, 04 Jun 2024 14:01:39 GMT

Hi Cloudera

Background: We run nifi as a standalone instance in a docker container in on all our stages, the statemanagement provider used by our instance is the local WAL backed provider.

Description: We observed that the flowfiles in front of one of our processor groups stopped to be ingested once in a while and it happens on all our stages without noticeable pattern. The flowfile concurrency policy of the processor group that stopped ingesting data is set to SINGLE_BATCH_PER_NODE and the outbound policy is set to BATCH_OUTPUT. The ingestion should continue since the processor group had already send out every flowfile in it, but it stopped.

There is only one brutal solution from our side(We need to delete the processor group and restore it from nifi registry again) and the occurrence of this issue had impacted our data ingestion.

As you can see in the screenshot that the processor group 'Delete before Insert' has no more flowfile to output but still it does not ingest the data queued in the input port

In the log file I found the following:

2024-06-03 11:34:01,772 TRACE [Timer-Driven Process Thread-15] o.apache.nifi.groups.StandardDataValve Will not allow data to flow into StandardProcessGroup[identifier=5eb0ad69-e8ed-3ba0-52da-af94fb9836cd,name=Delete before Insert] because Outbound Policy is Batch Output and valve is already open to allow data to flow out of group

Also in the diagnostics, I found the following for the 'Delete before Insert' processor group:

Process Group e6510a87-aa78-3268-1b11-3c310f0ad144, Name = Search and Delete existing reports(This is the parent processor group of the Delete before Insert)
Currently Have Data Flowing In: []
Currently Have Data Flowing Out: [StandardProcessGroup[identifier=5eb0ad69-e8ed-3ba0-52da-af94fb9836cd,name=Delete before Insert]]
Reason for Not allowing data to flow in:
    Data Valve is already allowing data to flow out of group:
        StandardProcessGroup[identifier=5eb0ad69-e8ed-3ba0-52da-af94fb9836cd,name=Delete before Insert]
Reason for Not allowing data to flow out:
    Output is Allowed:
        StandardProcessGroup[identifier=5eb0ad69-e8ed-3ba0-52da-af94fb9836cd,name=Delete before Insert]

Which is clearly not correct. since there are currently not data flowing out from the 'Delete before Insert' processor group

If you have any insight regarding this issue, it would be great if you car share it with me, also I think this could possibly be a nifi internal bug so I also opened an issue in nifi's jira page Here

Thank you for your time!

Re: Nifi: Flowfile stuck in front of a processor group when flowfile concurrency is set to single batch per node

DianaTorres — Tue, 04 Jun 2024 16:53:16 GMT

@yuanhao1999 Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @MattWho @mburgess who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Re: Nifi: Flowfile stuck in front of a processor group when flowfile concurrency is set to single batch per node

MattWho — Tue, 04 Jun 2024 17:02:05 GMT

@yuanhao1999
I see that you raised an Apache Jira for this same issue.
https://issues.apache.org/jira/browse/NIFI-13340
and that your issue is likely related to:
https://issues.apache.org/jira/browse/NIFI-13281

When you delete and re-create import the Process Group from NiFi-Registry, all your components will get new random UUIDs assigned to them. That effectively eliminates the stuck condition.

Where changes being made to process group configuration while FlowFile(s) were still queued in a connection within the Process Group?

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: Nifi: Flowfile stuck in front of a processor group when flowfile concurrency is set to single batch per node

yuanhao1999 — Wed, 05 Jun 2024 06:17:17 GMT

Hey Matt!

Thanks for your reply, so do you mean that this is indeed likely a nifi's internal bug instead of some mis-configuration?

Yuanhao

Re: Nifi: Flowfile stuck in front of a processor group when flowfile concurrency is set to single batch per node

yuanhao1999 — Tue, 11 Jun 2024 09:11:29 GMT

For those who's interested in this issue

The nifi dev team had relied and expect to fix this problem in 2.0.0-M4

Re: Nifi: Flowfile stuck in front of a processor group when flowfile concurrency is set to single batch per node

MattWho — Tue, 11 Jun 2024 15:22:46 GMT

Yes, i believe this to be a legitimate NiFi bug.