Problem kafka producer - WARN clients.NetworkClient: [Producer clientId=console-producer] Bootstrap broker

Question

we are using kafka 2.2.1 in cloudera manager with cdh6.3.2

when we try to run kafka producer:

kafka-console-producer --broker-list ip-172-31-13-30.us-east-2.compute.internal:9092,ip-172-31-12-151.us-east-2.compute.internal:9092,ip-172-31-14-136.us-east-2.compute.internal:9092 --topic test_topic

returns the errors:

22/07/08 12:36:16 WARN clients.NetworkClient: [Producer clientId=console-producer] Bootstrap broker ip-172-31-14-136.us-east-2.compute.internal:9092 (id: -3 rack: null) disconnected
22/07/08 12:36:16 WARN clients.NetworkClient: [Producer clientId=console-producer] Bootstrap broker ip-172-31-12-151.us-east-2.compute.internal:9092 (id: -2 rack: null) disconnected
22/07/08 12:36:16 WARN clients.NetworkClient: [Producer clientId=console-producer] Bootstrap broker ip-172-31-13-30.us-east-2.compute.internal:9092 (id: -1 rack: null) disconnected

when we check the logs of kafka brokers there is:

2022-07-08 12:36:17,365 INFO org.apache.kafka.common.network.Selector: [SocketServer brokerId=89] Failed authentication with /172.31.9.160 (Unexpected Kafka request of type METADATA during SASL handshake.)

2022-07-08 12:34:11,776 INFO org.apache.kafka.common.network.Selector: [SocketServer brokerId=88] Failed authentication with /172.31.9.160 (Unexpected Kafka request of type METADATA during SASL handshake.)

2022-07-08 12:34:11,505 INFO org.apache.kafka.common.network.Selector: [SocketServer brokerId=90] Failed authentication with /172.31.9.160 (Unexpected Kafka request of type METADATA during SASL handshake.)

We would like to know why this problem is occurring?

ps: we are managing to list the topics without problem

bgoerlitz · Accepted Answer

These errors on the client side usually indicate a security misconfiguration, and based on the broker log messages you sent, I would guess that you have Kerberos enabled for this cluster ("SASL handshake" suggests kerberos is expected). When anything other than PLAINTEXT (no security) is used, you will need to provide a client configuration file specifying the security properties.

Since the specific values of those properties would be dependent on your environment, I'd recommend that you review steps 7 and following in the Kafka Security - Enabling Kerberos Authentication documentation. Here is a link to that section

https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/kafka_security.html#kafka_kerberos

I hope that helps!

DianaTorres · Answer

@yagoaparecidoti Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.