Support Questions

Find answers, ask questions, and share your expertise

Kafka console consumer nor reading messages when used with bootstrap servers option

avatar

I am using kafka console consumer script to read messages from a kafka topic. but it is showing the following behavior.

If i use the zookeeper option, the consumer reads messages, whereas if i use bootstrap-server option i am not able to read messages. any thoughts.

using HDP 2.6

 bin/kafka-console-consumer.sh --bootstrap-server 172.16.3.196:6667 --topic test1 --from-beginning 
bin/kafka-console-consumer.sh --zookeeper 172.16.3.196:2181 --topic test1 --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
Hello
How are you
this is a test message
Hope this helps
11 REPLIES 11

avatar
Contributor

Just to make sure, could you please check if the Kafka broker is listening on 172.16.3.196 machine and port number is 6667.

avatar

@Kartik Ramalingam, yes it is listening

 netstat -na | grep 6667
tcp        0      0 172.16.3.196:6667       0.0.0.0:*               LISTEN
tcp        0      0 172.16.3.196:57076      172.16.3.196:6667       TIME_WAIT
tcp        0      0 172.16.3.196:6667       172.16.3.197:58774      ESTABLISHED
tcp        0      0 172.16.3.196:6667       172.16.3.197:39880      ESTABLISHED

avatar
Contributor

Is it a secured HDP cluster and what version of Kafka and Scala you are running on?

If your Kafka is not kerberized , lets try to command out Kerberos related stuff and give it a try. To fix them comment out Kerberos related commands from them on all brokers.

sed -i '/^export KAFKA_CLIENT_KERBEROS_PARAMS/s/^/# /' /usr/hdp/current/kafka-broker/bin/*.shgrep "export KAFKA_CLIENT_KERBEROS" /usr/hdp/current/kafka-broker/bin/*.sh       # to confirm

avatar
Contributor

Hi Arun, did you solved the issue?

I'm having the same problem when trying to use nifi processors ConsumeKafka and ConsumeKafka_0_10 to read messages from kafka 0.10.1 (HDP 2.6 without kerberos). I did tests with different parameter settings and the processors starts without error but never receives the messages. I had same results executing the kafka-console-consumer shell with bootstrap-server parameter even commenting out commands related to Kerberos.

Any suggestion?

avatar
Contributor

1. what version of kafka you are running?

2. what error you are getting in the console?

3. also use --new-consumer explicitely and share the errors after producing some messages while console consumer running.

avatar
Contributor

@Nikhil Silsarma I'm working with kafka 0.10.1 (HDP 2.6.3)

I don't receive any errors from the console consumer or the Nifi processor. Not even using the "--new-consumer" option.

Tell me if you need more details.

avatar
Contributor

Can you please share your kafka logs? Please enable log level to DEBUG n tools-log4j.

Wondering if your brokerid changed or live .

avatar
New Contributor

Hi,
I have the same issue. HDP 2.6.3
I have tried to run the following commands

WORKING:

bin/kafka-console-consumer.sh --zookeeper master.server:2181 --topic test --from-beginning
bin/kafka-simple-consumer-shell.sh --broker-list worker-2.server:6667,worker-3.server:6667,worker-4.server:6667 --topic test --partition 0

NOT WORKING:

bin/kafka-console-consumer.sh --bootstrap-server worker-2.server:6667 --new-consumer --topic test --from-beginning

I have configured log4j.rootLogger=DEBUG via ambari. But I can't see anything in the logs.

Can you help please?

avatar
New Contributor