Unable to push any messages to Kafka

[#1]

Hi there,

My question is quite similar to the

https://community.hortonworks.com/questions/40457/kafka-producer-giving-error-when-running-from-a-di...

and

https://community.hortonworks.com/questions/23775/unable-to-produce-message.html

But this fails at the very first step : while trying to sending messages to Kafka topic

I am using the below commands to start the producer:

bin/kafka-console-producer.sh --broker-list <HOST_FQDN>:6667  --topic test

This FQDN is copied from the $ hostname -f , also verified, the <HOST_FQDN> is the one matching in <KAFKA_BROKER_HOME>/config/server.properites

advertised.listeners=PLAINTEXTSASL://<HOST_FQDN_SAME_AS_HOSTNAME_F>:6667

Now when I start the server I see :

bin/kafka-console-producer.sh --broker-list <BROKER_FQDN>:6667  --topic test

Note : test is a valid topic created previously.

HI
[2016-09-07 18:10:15,713] WARN Fetching topic metadata with correlation id 0 for topics [Set(test)] from broker [BrokerEndPoint(0,<BROKER_FQDN>,6667)] failed (kafka.client.ClientUtils$)
java.io.EOFException
        at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:83)
        at kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:140)
        at kafka.network.BlockingChannel.receive(BlockingChannel.scala:131)
        at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:79)
        at kafka.producer.SyncProducer.kafka$producer$SyncProducer$doSend(SyncProducer.scala:76)
        at kafka.producer.SyncProducer.send(SyncProducer.scala:121)
        at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
        at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
        at kafka.producer.async.DefaultEventHandler$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:68)
        at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:79)
        at kafka.utils.Logging$class.swallowError(Logging.scala:106)
        at kafka.utils.CoreUtils$.swallowError(CoreUtils.scala:51)
        at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:68)
        at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105)
        at kafka.producer.async.ProducerSendThread$anonfun$processEvents$3.apply(ProducerSendThread.scala:88)
        at kafka.producer.async.ProducerSendThread$anonfun$processEvents$3.apply(ProducerSendThread.scala:68)
        at scala.collection.immutable.Stream.foreach(Stream.scala:547)
        at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67)
        at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:45)
[2016-09-07 18:10:15,716] ERROR fetching topic metadata for topics [Set(test)] from broker [ArrayBuffer(BrokerEndPoint(0,<BROKER_FQDN>,6667))] failed (kafka.utils.CoreUtils$)

Also checked kafka.out and server.log files, does not show any errors/exceptions

It would be really helpful if someone can help me understand missing bit.

Thanks,

SS

[#2 · Accepted solution]

Hi,

You have configured PLAINTEXTSASL port at Broker side. So we need to pass "--security-protocol PLAINTEXTSASL" option to kafka-console-producer.sh script. Also you need to pass required JAAS file/run kinit command.

refer below doc

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_secure-kafka-ambari/content/ch_secure-kaf...

[#3]

Thank you.