Kafka console consumer nor reading messages when used with bootstrap servers option

[#1]

I am using kafka console consumer script to read messages from a kafka topic. but it is showing the following behavior.

If i use the zookeeper option, the consumer reads messages, whereas if i use bootstrap-server option i am not able to read messages. any thoughts.

using HDP 2.6

 bin/kafka-console-consumer.sh --bootstrap-server 172.16.3.196:6667 --topic test1 --from-beginning 
bin/kafka-console-consumer.sh --zookeeper 172.16.3.196:2181 --topic test1 --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
Hello
How are you
this is a test message
Hope this helps

[#2]

Just to make sure, could you please check if the Kafka broker is listening on 172.16.3.196 machine and port number is 6667.

[#3]

@Kartik Ramalingam, yes it is listening

 netstat -na | grep 6667
tcp        0      0 172.16.3.196:6667       0.0.0.0:*               LISTEN
tcp        0      0 172.16.3.196:57076      172.16.3.196:6667       TIME_WAIT
tcp        0      0 172.16.3.196:6667       172.16.3.197:58774      ESTABLISHED
tcp        0      0 172.16.3.196:6667       172.16.3.197:39880      ESTABLISHED

[#4]

Is it a secured HDP cluster and what version of Kafka and Scala you are running on?

If your Kafka is not kerberized , lets try to command out Kerberos related stuff and give it a try. To fix them comment out Kerberos related commands from them on all brokers.

sed -i '/^export KAFKA_CLIENT_KERBEROS_PARAMS/s/^/# /' /usr/hdp/current/kafka-broker/bin/*.shgrep "export KAFKA_CLIENT_KERBEROS" /usr/hdp/current/kafka-broker/bin/*.sh       # to confirm

[#5]

Hi Arun, did you solved the issue?

I'm having the same problem when trying to use nifi processors ConsumeKafka and ConsumeKafka_0_10 to read messages from kafka 0.10.1 (HDP 2.6 without kerberos). I did tests with different parameter settings and the processors starts without error but never receives the messages. I had same results executing the kafka-console-consumer shell with bootstrap-server parameter even commenting out commands related to Kerberos.

Any suggestion?

[#6]

1. what version of kafka you are running?

2. what error you are getting in the console?

3. also use --new-consumer explicitely and share the errors after producing some messages while console consumer running.

[#7]

@Nikhil Silsarma I'm working with kafka 0.10.1 (HDP 2.6.3)

I don't receive any errors from the console consumer or the Nifi processor. Not even using the "--new-consumer" option.

Tell me if you need more details.

[#8]

Can you please share your kafka logs? Please enable log level to DEBUG n tools-log4j.

Wondering if your brokerid changed or live .

[#9]

Hi,
I have the same issue. HDP 2.6.3
I have tried to run the following commands

WORKING:

bin/kafka-console-consumer.sh --zookeeper master.server:2181 --topic test --from-beginning
bin/kafka-simple-consumer-shell.sh --broker-list worker-2.server:6667,worker-3.server:6667,worker-4.server:6667 --topic test --partition 0

NOT WORKING:

bin/kafka-console-consumer.sh --bootstrap-server worker-2.server:6667 --new-consumer --topic test --from-beginning

I have configured log4j.rootLogger=DEBUG via ambari. But I can't see anything in the logs.

Can you help please?

[#10]

@Nikhil Silsarma