kafka-console-consumer not reading messages

[#1]

I'm using HDP 2.3.4 with kafka 0.9

I just started to use kafka referring to this document, but having problem with the kafka-console-consumer.

below are the commands used to create a topic and publish messages

bin/kafka-topics.sh --zookeeper <serverX>:2181 --create --topic test2 --partitions 2 --replication-factor 1

bin/kafka-console-producer.sh --broker-list <brokerX>:6667 --topic test2 --security-protocol SASL_PLAINTEXT

both commands worked well.

Tried kafka simple consumer, and worked well, message were read and displayed

bin/kafksimple-consumer-shell.sh --broker-list <brokerX>:6667 --topic test2 --security-protocol SASL_PLAINTEXT --partition 0

bin/kafksimple-consumer-shell.sh --broker-list <brokerX>:6667 --topic test2 --security-protocol SASL_PLAINTEXT --partition 1

but I'm not able to get kafka-console-consumer work.

Tried below command first, but hitting exceptions:

bin/kafka-console-consumer.sh --zookeeper <serverX>:2181 --topic test2 --from-beginning --security-protocol SASL_PLAINTEXT

WARN [console-consumer-89015_xxx-1475146648772-75b1a67a-leader-finder-thread], Failed to find leader for Set([test2,0], [test2,1]) (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread) kafka.common.BrokerEndPointNotAvailableException: End point SASL_PLAINTEXT not found for broker 1002

some online post suggested to use new-consumer option for kafka 0.9, so tried the command below:

bin/kafka-console-consumer.sh --zookeeper <serverX>:2181 --topic test2 --bootstrap-server <serverY>:9092 --new-consumer --security-protocol SASL_PLAINTEXT

consumer started without error, but no messages were read and displayed.

I'm not sure what went wrong, any suggestions? Thanks a lot!

[#2]

If you are not using Kerberos, can you open bin/kafka-console-consumer.sh in an editor and comment out Kerberos related "export" command from the script. In my related article, at the bottom, you can find a sed command how to comment out those commands in all kafka scripts.

[#3]

Hi @pp z

• Check whether console consumer works with --bootstrap-server = broker instead of --zookeeper.

• Check advertised listener in server.properties. Is it of the form PLAINTEXTSASL://host:port?
• Check whether security_protocol = PLAINTEXTSASL exists in server.properties.
• Are you getting the exception: WARN SASL configuration failed:javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file. If yes, then modify kafka_client_jaas.conf to include a Client section, you'll most likely get that in kafka_jaas.conf. If not, then ignore this.
• Use --security-protocol PLAINTEXTSASL instead of SASL_PLAINTEXT.
• Check whether you're able to start a authenticated connection to zookeeper-client. If not, give the path of conf file that has client section as a JVM param. For example: export JVMFLAGS="-Djava.security.auth.login.config= /usr/hdp/2.6.1.0-129/kafka/conf/kafka_client_jaas.conf"

Try these and let me know.

Thanks

[#4]

--security-protocol PLAINTEXTSASL solved it