Reply
Explorer
Posts: 6
Registered: ‎08-06-2017

Kafka Producer Connection Pool is required or not

[ Edited ]

Hi,

I am developing a plugin for accessing multiple brokers and topic parallelly. 

 

Does it make sense to perform producer/consumer connection pooling of kafka clients?

 

Does kafka internally maintain a list of connection objects initialized and ready to use?

 

We'd like to minimize time of connection creation, so that there is no additional overhead when it comes to send/receive messages.

 

Please share your feedback.

 

Regards

Gopa

Cloudera Employee
Posts: 30
Registered: ‎04-05-2016

Re: Kafka Producer Connection Pool is required or not

Gopa,

There is no need to pool the client connections as KafkaProducer and KafkaConsumer usually keeps the broker connection open and allow you to produce/consume the events bypassing the connection creation overhead. The connection can be closed by calling close() method of each clients. 

 

Also as a side-note, the producer is thread safe and sharing a single producer instance across threads will generally be faster than having multiple instances. However the Kafka consumer is NOT thread-safe. All network I/O happens in the thread of the application making the call. It is the responsibility of the user to ensure that multi-threaded access is properly synchronized.

 

Explorer
Posts: 6
Registered: ‎08-06-2017

Re: Kafka Producer Connection Pool is required or not

Thank you Umesh for your quick reply. 

 

Now i am facing some issues while trying to access HBase from Spark code. Details are in below point

 

https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/HBase-access-in-Spark-code-in-a-Ke...

 

It would be great if you can give suggestion on this topic also.

Thanks

Gopa

 

Announcements

Currently incubating in Cloudera Labs:

Envelope
HTrace
Ibis
Impyla
Livy
Oryx
Phoenix
Spark Runner for Beam SDK
Time Series for Spark
YCSB