Our production data pipeline is consuming messages from kafka topic with 20 partitions. The messages are produced at the rate of 5000 messages per second .The kafka consumer in the data pipeline is configured to 7 concurrent task in 3 node cluster. The consumeKafka is able to consume only 3000 messages per second.
Question: why consume kafka not able to consume at the same rate of producer? We are seeking the approaches or options can make the status of Kafka partitions with same topic and queue count more balanced.
Already a User?
Don't have an account?
Coming from Hortonworks? Activate your account here
