Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

listentTCP, Publishkafka throughput performance and latency

avatar
Expert Contributor

Hi All,

Thanks a lot to this awesome community.

I was reading this blog by @Bryan Bende, and in the last paragraph, as per my interpretation in order to have better performance we should have 1 large flowfile as in output from listenTCP which can be achieved by using configuring "Max Batch size I set it to "100 000" . this will collect 100000 messages in a flowfile and input it to PublishKafka and in the Publish kafka the "Message Demaractor" is set to new line (shift+enter) this will send each message from the flow file thus increasing throughput

Am I correct in my understanding?

Thanks

Dheeru

1 ACCEPTED SOLUTION

avatar
Master Guru

Yes this is correct!

It is hard to say what batch size is the best, but as long as you are batching together at least a couple of thousand messages, it will be significantly better than 1 message per flow file. So maybe start with 10,000 and tune from there.

View solution in original post

2 REPLIES 2

avatar
Master Guru

Yes this is correct!

It is hard to say what batch size is the best, but as long as you are batching together at least a couple of thousand messages, it will be significantly better than 1 message per flow file. So maybe start with 10,000 and tune from there.

avatar
Expert Contributor

@Bryan Bende Aweosme Thanks and appreciate it