question Re: Data is becoming stuck after Input Port in Nifi in Archives of Support Questions (Read Only)

Data is becoming stuck after Input Port in Nifi

elloyd — Sun, 18 Aug 2019 01:46:12 GMT

We currently have a setup of 3 Nifis in a cluster. I have MiniFi on a sending server sending to the Input port in the screenshot called LoggerMiniFi2.

The data always gets stuck right after the Input Port. By stuck I mean it queues up and then just sits there, never empties. We cannot just drop the data, its important data so setting an expiration time doesn't make sense. You can see in the screenshot below under Status History that the data has sat in there with no bytes out for almost 15 minutes.

Can anyone provide any troubleshooting tips on this, perhaps a cause? Note, I have also tried removing the SplitText processor and just connecting the Input Port to the ExtractText and it still just stalled out right after the Input Port. I'm at a loss of what could be causing this.

Note, I have put the SplitText configs down below just for reference as well.

Re: Data is becoming stuck after Input Port in Nifi

bbende — Thu, 22 Jun 2017 23:56:49 GMT

Your SplitText processor is scheduled to run on Primary Node only which doesn't seem right. MiNiFi would send data to all nodes.

Most likely the flow files that are sitting there are not on the primary node, which you can determine by doing a List Queue on that connection and looking at the host column on the right.

Re: Data is becoming stuck after Input Port in Nifi

elloyd — Fri, 23 Jun 2017 00:01:36 GMT

But if I say "run on all nodes" doesn't that duplicate the data output ?? Or am I misunderstanding...

Re: Data is becoming stuck after Input Port in Nifi

elloyd — Fri, 23 Jun 2017 00:04:06 GMT

That was it! I don't understand the difference between setting it to Primary Node from All Nodes. I thought it would duplicate all the output data.

Re: Data is becoming stuck after Input Port in Nifi

bbende — Fri, 23 Jun 2017 00:04:35 GMT

No, you have a 3 node cluster, lets say node #1 is primary node... MiNiFi is sending data to all nodes so the data is already divided across all the nodes, but you are only scheduled to process it on node #1, so now data on nodes #2 and #3 will just sit there and never get processed.

Re: Data is becoming stuck after Input Port in Nifi

bbende — Fri, 23 Jun 2017 00:05:22 GMT

Generally you only want Primary Node only for a source processor like ListHDFS where you only want to perform the listing one time.

Re: Data is becoming stuck after Input Port in Nifi

elloyd — Fri, 23 Jun 2017 00:26:57 GMT

Awesome thanks again !