Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

ConsumeKafka performance issue

ConsumeKafka performance issue

New Contributor

Our production data pipeline is consuming messages from kafka topic with 20 partitions. The messages are produced at the rate of 5000 messages per second .The kafka consumer in the data pipeline is configured to 7 concurrent task in 3 node cluster. The consumeKafka is able to consume only 3000 messages per second.

Question: why consume kafka not able to consume at the same rate of producer? We are seeking the approaches or options can make the status of Kafka partitions with same topic and queue count more balanced.

93507-piepline-settings.png

93506-nodes-tab-in-cluster.png

93505-timer-driven-threads.png