PutTCP-ListenTCP NiFi to NiFi issue

[#1]

I am unable to get TCP nifi to nifi to work.

I have a put TCP processor that puts files to a port and host, but when the receiving host receives the TCP via listen TCP it receives a TON of small files that are not the original flowfile.

Is listen TCP not putting the packets back together into a flowfile? or am I doing something wrong here?

Attached are pics of my configs and setup.

Thank you!

[#2 · Accepted solution]

@John T

The GenerateFlowFile processor is capable of creating multi-line content in a FlowFile. Even after passing that content through the Base64EncodeContent Processor the content will remain multi-lined.

The putTCP processor is then pushing the multi-line content in to the internal socket buffer of the ListenTCP processor. The ListenTCP processor will then read those lines and create new FlowFiles in which they will be placed based on the configured batch size. In your case you have a batch size of only 1, so each created FlowFile will only contain a single line each. This would explain why you see so many little FlowFIles being generated on your receiving side.

If you are just trying to sedn a FlowFile from one NiFi to another, why not use NiFi Site-To-SIte or the PostHTTP --> ListenHTTP processors (These two methods can send actual "FlowFiles" between NiFi instances)?

Thank you,

Matt

If you found this answer addressed your question, please take a moment to click the "accept" link below.

[#3]

@Matt Clarke

Hey Matt,

Thanks for the quick feedback! So I can just change the batch size on the listen tcp to a really large number and regaurdless of the actual number of new lines it will package back together a full flowfile?

I really need to do TCP to TCP because there is a middle system I will have to send my stream through, that will inspect the stream and TCP is the only way to do so.

Any other ideas?

John

[#4]

@John T

First, let's make sure we are on the same page with terminology.

FlowFile --> A FlowFile is something very specific to NiFi. Each "FlowFile" consists of two parts, the FlowFile Attributes/Metadata and the FlowFile content.

In most cases, processors do not send "FlowFiles", but only send the FlowFile's content. The postHTTP has a option property to (send as FlowFile) which will allow it to send a complete FLowFile to a ListenHTTP processor running on another NiFi instance. NiFi's Site-To-Site capability (uses Remote Process Group and Remote input/output ports) to send "FlowFiles" between Nifi instances. The only other method to send a "FlowFile" between NiFi instances is to use the MergeContent processor configured with a "Merge Format" that will package up the complete FlowFile (FlowFile Stream or FlowFile tar) in the content. This packaged FlowFile can then be sent to another NiFi and the UnPackContent processor can be used to separate the original FlowFile Content from FlowFile Attributes (load original content on content repo and original Attributes in to FlowFile repo)

Simply increasing the batch size to a large value here will not guarantee the original content is re-created in destination FlowFile. Since content is streaming in to the socket buffer and NiFi's ListenTCP processor simply calls the channel Reader to ingest from the buffer line-by-line, you may end up with lines from one source's content in the same FlowFile as lines of content from a different source File.

For a little more info on the NiFi ListenTCP based processor:

https://community.hortonworks.com/articles/30424/optimizing-performance-of-apache-nifis-network-lis....

If you end goal is to send complete FlowFile content, consider using a different transfer method from putTCP and listenTCP. The listenTCP processor is designed in such a way it assumes each line being read from teh socket buffer is its own incoming message.

Thank you,

Matt

[#5]

@John T

Was I able to address your question here on how this putTCP --> listenTCP flow is working?

[#6]

Yes thank you matt!