listentTCP, Publishkafka throughput performance and latency

[#1]

Hi All,

Thanks a lot to this awesome community.

I was reading this blog by @Bryan Bende, and in the last paragraph, as per my interpretation in order to have better performance we should have 1 large flowfile as in output from listenTCP which can be achieved by using configuring "Max Batch size I set it to "100 000" . this will collect 100000 messages in a flowfile and input it to PublishKafka and in the Publish kafka the "Message Demaractor" is set to new line (shift+enter) this will send each message from the flow file thus increasing throughput

Am I correct in my understanding?

Thanks

Dheeru

[#2 · Accepted solution]

Yes this is correct!

It is hard to say what batch size is the best, but as long as you are batching together at least a couple of thousand messages, it will be significantly better than 1 message per flow file. So maybe start with 10,000 and tune from there.

[#3]

@Bryan Bende Aweosme Thanks and appreciate it