I am using kafka console consumer script to read messages from a kafka topic. but it is showing the following behavior.
If i use the zookeeper option, the consumer reads messages, whereas if i use bootstrap-server option i am not able to read messages. any thoughts.
using HDP 2.6
bin/kafka-console-consumer.sh --bootstrap-server 172.16.3.196:6667 --topic test1 --from-beginning bin/kafka-console-consumer.sh --zookeeper 172.16.3.196:2181 --topic test1 --from-beginning Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper]. Hello How are you this is a test message Hope this helps
Just to make sure, could you please check if the Kafka broker is listening on 172.16.3.196 machine and port number is 6667.
@Kartik Ramalingam, yes it is listening
netstat -na | grep 6667 tcp 0 0 172.16.3.196:6667 0.0.0.0:* LISTEN tcp 0 0 172.16.3.196:57076 172.16.3.196:6667 TIME_WAIT tcp 0 0 172.16.3.196:6667 172.16.3.197:58774 ESTABLISHED tcp 0 0 172.16.3.196:6667 172.16.3.197:39880 ESTABLISHED
Is it a secured HDP cluster and what version of Kafka and Scala you are running on?
If your Kafka is not kerberized , lets try to command out Kerberos related stuff and give it a try. To fix them comment out Kerberos related commands from them on all brokers.
sed -i '/^export KAFKA_CLIENT_KERBEROS_PARAMS/s/^/# /' /usr/hdp/current/kafka-broker/bin/*.shgrep "export KAFKA_CLIENT_KERBEROS" /usr/hdp/current/kafka-broker/bin/*.sh # to confirm
Hi Arun, did you solved the issue?
I'm having the same problem when trying to use nifi processors ConsumeKafka and ConsumeKafka_0_10 to read messages from kafka 0.10.1 (HDP 2.6 without kerberos). I did tests with different parameter settings and the processors starts without error but never receives the messages. I had same results executing the kafka-console-consumer shell with bootstrap-server parameter even commenting out commands related to Kerberos.
1. what version of kafka you are running?
2. what error you are getting in the console?
3. also use --new-consumer explicitely and share the errors after producing some messages while console consumer running.
@Nikhil Silsarma I'm working with kafka 0.10.1 (HDP 2.6.3)
I don't receive any errors from the console consumer or the Nifi processor. Not even using the "--new-consumer" option.
Tell me if you need more details.
Can you please share your kafka logs? Please enable log level to DEBUG n tools-log4j.
Wondering if your brokerid changed or live .
I have the same issue. HDP 2.6.3
I have tried to run the following commands
bin/kafka-console-consumer.sh --zookeeper master.server:2181 --topic test --from-beginning bin/kafka-simple-consumer-shell.sh --broker-list worker-2.server:6667,worker-3.server:6667,worker-4.server:6667 --topic test --partition 0
bin/kafka-console-consumer.sh --bootstrap-server worker-2.server:6667 --new-consumer --topic test --from-beginning
I have configured log4j.rootLogger=DEBUG via ambari. But I can't see anything in the logs.
Can you help please?