Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

Kafka console consumer nor reading messages when used with bootstrap servers option

Highlighted

Kafka console consumer nor reading messages when used with bootstrap servers option

I am using kafka console consumer script to read messages from a kafka topic. but it is showing the following behavior.

If i use the zookeeper option, the consumer reads messages, whereas if i use bootstrap-server option i am not able to read messages. any thoughts.

using HDP 2.6

 bin/kafka-console-consumer.sh --bootstrap-server 172.16.3.196:6667 --topic test1 --from-beginning 
bin/kafka-console-consumer.sh --zookeeper 172.16.3.196:2181 --topic test1 --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
Hello
How are you
this is a test message
Hope this helps
10 REPLIES 10

Re: Kafka console consumer nor reading messages when used with bootstrap servers option

New Contributor

Just to make sure, could you please check if the Kafka broker is listening on 172.16.3.196 machine and port number is 6667.

Re: Kafka console consumer nor reading messages when used with bootstrap servers option

@Kartik Ramalingam, yes it is listening

 netstat -na | grep 6667
tcp        0      0 172.16.3.196:6667       0.0.0.0:*               LISTEN
tcp        0      0 172.16.3.196:57076      172.16.3.196:6667       TIME_WAIT
tcp        0      0 172.16.3.196:6667       172.16.3.197:58774      ESTABLISHED
tcp        0      0 172.16.3.196:6667       172.16.3.197:39880      ESTABLISHED

Re: Kafka console consumer nor reading messages when used with bootstrap servers option

New Contributor

Is it a secured HDP cluster and what version of Kafka and Scala you are running on?

If your Kafka is not kerberized , lets try to command out Kerberos related stuff and give it a try. To fix them comment out Kerberos related commands from them on all brokers.

sed -i '/^export KAFKA_CLIENT_KERBEROS_PARAMS/s/^/# /' /usr/hdp/current/kafka-broker/bin/*.shgrep "export KAFKA_CLIENT_KERBEROS" /usr/hdp/current/kafka-broker/bin/*.sh       # to confirm

Re: Kafka console consumer nor reading messages when used with bootstrap servers option

New Contributor

Hi Arun, did you solved the issue?

I'm having the same problem when trying to use nifi processors ConsumeKafka and ConsumeKafka_0_10 to read messages from kafka 0.10.1 (HDP 2.6 without kerberos). I did tests with different parameter settings and the processors starts without error but never receives the messages. I had same results executing the kafka-console-consumer shell with bootstrap-server parameter even commenting out commands related to Kerberos.

Any suggestion?

Re: Kafka console consumer nor reading messages when used with bootstrap servers option

New Contributor

1. what version of kafka you are running?

2. what error you are getting in the console?

3. also use --new-consumer explicitely and share the errors after producing some messages while console consumer running.

Re: Kafka console consumer nor reading messages when used with bootstrap servers option

New Contributor

@Nikhil Silsarma I'm working with kafka 0.10.1 (HDP 2.6.3)

I don't receive any errors from the console consumer or the Nifi processor. Not even using the "--new-consumer" option.

Tell me if you need more details.

Re: Kafka console consumer nor reading messages when used with bootstrap servers option

New Contributor

Can you please share your kafka logs? Please enable log level to DEBUG n tools-log4j.

Wondering if your brokerid changed or live .

Re: Kafka console consumer nor reading messages when used with bootstrap servers option

New Contributor

Hi,
I have the same issue. HDP 2.6.3
I have tried to run the following commands

WORKING:

bin/kafka-console-consumer.sh --zookeeper master.server:2181 --topic test --from-beginning
bin/kafka-simple-consumer-shell.sh --broker-list worker-2.server:6667,worker-3.server:6667,worker-4.server:6667 --topic test --partition 0

NOT WORKING:

bin/kafka-console-consumer.sh --bootstrap-server worker-2.server:6667 --new-consumer --topic test --from-beginning

I have configured log4j.rootLogger=DEBUG via ambari. But I can't see anything in the logs.

Can you help please?

Re: Kafka console consumer nor reading messages when used with bootstrap servers option

New Contributor