Support Questions
Find answers, ask questions, and share your expertise

Kafka console consumer nor reading messages when used with bootstrap servers option

I am using kafka console consumer script to read messages from a kafka topic. but it is showing the following behavior.

If i use the zookeeper option, the consumer reads messages, whereas if i use bootstrap-server option i am not able to read messages. any thoughts.

using HDP 2.6

 bin/ --bootstrap-server --topic test1 --from-beginning 
bin/ --zookeeper --topic test1 --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
How are you
this is a test message
Hope this helps

Cloudera Employee

Just to make sure, could you please check if the Kafka broker is listening on machine and port number is 6667.

@Kartik Ramalingam, yes it is listening

 netstat -na | grep 6667
tcp        0      0*               LISTEN
tcp        0      0       TIME_WAIT
tcp        0      0      ESTABLISHED
tcp        0      0      ESTABLISHED

Cloudera Employee

Is it a secured HDP cluster and what version of Kafka and Scala you are running on?

If your Kafka is not kerberized , lets try to command out Kerberos related stuff and give it a try. To fix them comment out Kerberos related commands from them on all brokers.

sed -i '/^export KAFKA_CLIENT_KERBEROS_PARAMS/s/^/# /' /usr/hdp/current/kafka-broker/bin/*.shgrep "export KAFKA_CLIENT_KERBEROS" /usr/hdp/current/kafka-broker/bin/*.sh       # to confirm

Hi Arun, did you solved the issue?

I'm having the same problem when trying to use nifi processors ConsumeKafka and ConsumeKafka_0_10 to read messages from kafka 0.10.1 (HDP 2.6 without kerberos). I did tests with different parameter settings and the processors starts without error but never receives the messages. I had same results executing the kafka-console-consumer shell with bootstrap-server parameter even commenting out commands related to Kerberos.

Any suggestion?

1. what version of kafka you are running?

2. what error you are getting in the console?

3. also use --new-consumer explicitely and share the errors after producing some messages while console consumer running.

@Nikhil Silsarma I'm working with kafka 0.10.1 (HDP 2.6.3)

I don't receive any errors from the console consumer or the Nifi processor. Not even using the "--new-consumer" option.

Tell me if you need more details.

Can you please share your kafka logs? Please enable log level to DEBUG n tools-log4j.

Wondering if your brokerid changed or live .

New Contributor

I have the same issue. HDP 2.6.3
I have tried to run the following commands


bin/ --zookeeper master.server:2181 --topic test --from-beginning
bin/ --broker-list worker-2.server:6667,worker-3.server:6667,worker-4.server:6667 --topic test --partition 0


bin/ --bootstrap-server worker-2.server:6667 --new-consumer --topic test --from-beginning

I have configured log4j.rootLogger=DEBUG via ambari. But I can't see anything in the logs.

Can you help please?

New Contributor

New Contributor

I got it working.

It seems there were some phantom borkers in zookeeper because I reconfigured the location of the kafka nodes after installation.

Removing the brokers from zookeeper and restarting the kafka nodes solved the issue:

bin/zookeeper-shell <zk-host>:2181
rmr /borker



New Contributor

you saved my day!

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.