kafka producer not accepecting messages

[#1]

Hi 

I am using cloudera cdh 5.10 with kafka 

When I run simple command line consumer I get the following error.

ERROR internals.ErrorLoggingCallback: Error when sending message to topic t1 with key: null, value: 2 bytes with error:
org.apache.kafka.common.errors.TimeoutException: Batch containing 1 record(s) expired due to timeout while requesting metadata from brokers for t1-0

Also when I run the consumer I get an endless loop below messages

17/03/26 19:26:24 INFO utils.VerifiableProperties: Verifying properties
17/03/26 19:26:24 INFO utils.VerifiableProperties: Property client.id is overridden to console-consumer-17838
17/03/26 19:26:24 INFO utils.VerifiableProperties: Property metadata.broker.list is overridden to 0.0.0.0:9092
17/03/26 19:26:24 INFO utils.VerifiableProperties: Property request.timeout.ms is overridden to 30000
17/03/26 19:26:24 INFO client.ClientUtils$: Fetching metadata from broker BrokerEndPoint(100,0.0.0.0,9092) with correlation id 14 for 1 topic(s) Set(t1)
17/03/26 19:26:24 INFO producer.SyncProducer: Connected to 0.0.0.0:9092 for producing
17/03/26 19:26:24 INFO producer.SyncProducer: Disconnecting from 0.0.0.0:9092
17/03/26 19:26:24 INFO consumer.ConsumerFetcherManager: [ConsumerFetcherManager-1490556381486] Added fetcher for partitions ArrayBuffer()
17/03/26 19:26:25 INFO utils.VerifiableProperties: Verifying properties
17/03/26 19:26:25 INFO utils.VerifiableProperties: Property client.id is overridden to console-consumer-17838
17/03/26 19:26:25 INFO utils.VerifiableProperties: Property metadata.broker.list is overridden to 0.0.0.0:9092
17/03/26 19:26:25 INFO utils.VerifiableProperties: Property request.timeout.ms is overridden to 30000
17/03/26 19:26:25 INFO client.ClientUtils$: Fetching metadata from broker BrokerEndPoint(100,0.0.0.0,9092) with correlation id 15 for 1 topic(s) Set(t1)
17/03/26 19:26:25 INFO producer.SyncProducer: Connected to 0.0.0.0:9092 for producing
17/03/26 19:26:25 INFO producer.SyncProducer: Disconnecting from 0.0.0.0:9092
17/03/26 19:26:25 INFO consumer.ConsumerFetcherManager: [ConsumerFetcherManager-1490556381486] Added fetcher for partitions ArrayBuffer()
17/03/26 19:26:25 INFO utils.VerifiableProperties: Verifying properties
17/03/26 19:26:25 INFO utils.VerifiableProperties: Property client.id is overridden to console-consumer-17838
17/03/26 19:26:25 INFO utils.VerifiableProperties: Property metadata.broker.list is overridden to 0.0.0.0:9092
17/03/26 19:26:25 INFO utils.VerifiableProperties: Property request.timeout.ms is overridden to 30000

Any Idea what configuration mistake is causing this issue

[#2]

Is zookeeper up and running properly? - try connecting to it using the zookeeper client to verify.

Try stopping both services, and then restarting them, zookeeper first.

Also for the consumer, try adding --from-beginning if you continue to have problems with it and not the producer.

[#3]

Thanks for the quick reply.

As per cloudera Manager the zookeeper appears to be running fine,when I conenct to the zookeepr client.

I get the following msg

2017-03-27 06:44:29,468 [myid:] - WARN [main-SendThread(localhost:2181):ClientCnxn$SendThread@1102] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

But it eventually connect to zookeeper,I am able execute commands like "ls"

 I am able to create topics,list and describe them but  each time I run the commands I get the following warning message

17/03/27 06:54:36 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
17/03/27 06:54:37 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
17/03/27 06:54:37 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:42156, server: localhost/127.0.0.1:2181
17/03/27 06:54:37 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x35b031cdeed10a7, negotiated timeout = 30000

[#4]

Well I'd call this progress I guess... seems like we have a zookeeper issue.

Have you tried restarting the services?

[#5]

This appears to be your problem:

17/03/26 19:26:24 INFO utils.VerifiableProperties: Property metadata.broker.list is overridden to 0.0.0.0:9092

Are you setting metadata.broker.list in your kafka.properties on the brokers?

What is your LISTENERS property set to on the brokers?

-pd

[#6]

Hey Thanks for your reply

@tseader I did restart it but did not help.Was getting the same error.

I had a look at the logs and found the following error message

kafka.common.InconsistentBrokerIdException: Configured broker.id 201 doesn't match stored broker.id 100 in meta.properties. If you moved your data, make sure your configured broker.id matches. If you intend to create a new broker, you should remove all data in your data directories (log.dirs).
at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:648)
at kafka.server.KafkaServer.startup(KafkaServer.scala:187)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at com.cloudera.kafka.wrap.Kafka$.main(Kafka.scala:76)
at com.cloudera.kafka.wrap.Kafka.main(Kafka.scala)
2017-03-27 14:55:46,069 INFO kafka.server.KafkaServer: shutting down
2017-03-27 14:55:46,072 INFO kafka.log.LogManager: Shutting down.
2017-03-27 14:55:46,073 INFO kafka.log.LogCleaner: Shutting down the log cleaner.

@pdvorak 

So my listeners are set

listeners=PLAINTEXT://0.0.0.0:9092

As per the above error message it looks like there is an issue with meta.properties ,I tracked down the file and and changed the broker id in the file to 201.

It is working fine now.I am able send message and recieve them via consumer

But is there anyway to change the the broker id in  meta.properties from cloudera manager

[#7]

meta.properties was likely left over from a previous instance of the kafka broker. Did you remove the broker role instances and recreate them on the same machine? When that happens, the broker.id changes and kafka won't start properly as a safety measure. A good rule of thumb is if you need to remove the role instances and recreate, then delete everything from the kafka data dir (unless you need to use the partitions), in which case, you'll have to update that meta.properties file manually.
-pd