Kafka 0.9 new-producer on kerberized HDP 2.4

[#1]

I am trying to run kafka-console-producer.sh script on a kerberized HDP 2.4 sandbox with Kafka 0.9. I have created a topic and provided user permissions to the topic.

Below is the command I am using to start the producer:

./kafka-console-producer.sh --topic test-topic --broker-list sandbox.hortonworks.com:6667 --security-protocol PLAINTEXTSASL --new-producer

Below is the error I get when I try to produce something:

ERROR Error when sending message to topic test-topic with key: null, value: 8 bytes with error: Failed to update metadata after 60000 ms. (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)

It works fine when I don't use the new-producer, but I need to use the new producer API.

[#2 · Accepted solution]

For new producer api, try increasing the metadata.fetch.timeout.ms in producer config and also socket.timeout.ms

[#3]

I changed the above properties to 600000 ms, still the issue exists.

[#4]

What is your config/server.properties and config/producer.properties? Anything in zookeeper logs?

[#5]

We had same issue and increasing time out values worked.