kafka-console-producer not working in HDP 2.5/Kafka 0.10

[#1]

I have installed Kafka in HDP 2.5 cluster.

bin/kafka-console-producer.sh seems to get stuck and doesn't produce a test message. I have tried the following command, none of them seems to work

usr/hdp/current/kafka-broker$ bin/kafka-console-producer.sh --broker-list ip-172-31-103-20.us-west-2.compute.internal:6667 --topic test


usr/hdp/current/kafka-broker$ bin/kafka-console-producer.sh --broker-list localhost:6667 --topic test


usr/hdp/current/kafka-broker$ bin/kafka-console-producer.sh --broker-list ip-172-31-103-20.us-west-2.compute.internal:6667 --topic test --security-protocol PLAINTEXT

I have verified that the listener property in /etc/kafka/conf/server.properties on the broker host is correct (listeners=PLAINTEXT://ip-172-31-103-20.us-west-2.compute.internal:6667)

I have also tried uncommenting the following three lines in kafka-console-producer.sh script but that doesn't help either

#KAFKA_JAAS_CONF=$KAFKA_HOME/config/kafka_jaas.conf
#if [ -f $KAFKA_JAAS_CONF ]; then
#    export KAFKA_CLIENT_KERBEROS_PARAMS="-Djava.security.auth.login.config=$KAFKA_HOME/config/kafka_client_jaas.conf"
#fi

Could someone help me debug this issue?

[#2 · Accepted solution]

Hi Anurag,

Could you please try sending a test message after running the producer and also run a consumer and provide the output.

For example:

1. usr/hdp/current/kafka-broker$ bin/kafka-console-producer.sh --broker-list ip-172-31-103-20.us-west-2.compute.internal:6667--topic test
2. Send any test message like "Hello world"
3. Run consumer for example: usr/hdp/current/kafka-broker$ kafka-console-consumer.sh --zookeeper <zookeeper_host>:2181 --topic test --from-beginning
4. Provide the output. The expected behavior is that the consumer will consume the message "Hello World" and you should be able to see it in the console.

[#3]

Thanks @dbains

My bad. I was certainly under the assumption that `kafka-console-producer.sh` itself produce a test message. Thanks for clarifying that's not the case. I typed in the message and verified that it has been received by the consumer.

[#4]

Hi Anuraj,

No problem, I am glad it is working now!

Regards,

Deepna

[#5]

@dbains I have installed Kafka on HDF cluster and trying to create topics using kafka-topics.sh and also use the kafka-console-produce.sh but bash keeps telling me command not found !!!