Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

flume is not writing any messages to Kakfa using KafkaSink

flume is not writing any messages to Kakfa using KafkaSink

Hi All,

Flume is not writing any messages to Kafka topic using KafkaSink, but works when other sinks are used.

HDP 2.6.1

Below is the kafkaSink configuration.

testagent2.sinks.kafkasink2.channel	 	 = memchannel2
testagent2.sinks.kafkasink2.type		 = org.apache.flume.sink.kafka.KafkaSink
testagent2.sinks.kafkasink2.brokerList 		 = brokerServer:portNo
testagent2.sinks.kafkasink2.zookeeperConnect	 = ZookeeperServer:portNo
testagent2.sinks.kafkasink2.topic		 = testtopic2
testagent2.sinks.kafkasink2.batchSize		 = 100
testagent2.sinks.kafkasink2.requiredAcks 	 = 1
testagent2.sinks.kafkasink2.kafka.rebalance.max.retries 	= 40
testagent2.sinks.kafkasink2.kafka.rebalance.backoff.ms 		= 5000
testagent2.sinks.kafkasink2.kafka.zookeeper.session.timeout.ms	= 6000

Please note that Source and Channel configs are working without any issues when other sinks are being used.

Facing issue only with Kafka Sink. Also flume is not throwing any error in the logs and I can independently write messages to Kafka topics by other means.

Has anyone faced this issue ?

Thanks.

Don't have an account?
Coming from Hortonworks? Activate your account here