KAFKA CONSUMER NOT CONSUMING MESSAGES - Cloudera quickstart VM

[#1]

Hello Sir,

After installaing kafka in the cloudera manager i have tested the below code,but the consumer not able to consume those messages.While installing kafka in the cloudera manager,i had given the below two configurations as follows:

Destination Broker List ===> quickstart.cloudera:9092

bootstrap.servers

Source Broker List===> quickstart.cloudera:9092

source.bootstrap.servers

• Java Heap Size of Broker (broker_max_heap_size) =“256”
• Advertised Host (advertised.host.name) = “quickstart.cloudera”
• Inter Broker Protocol = “PLAINTEXT”

kafka-topics --zookeeper quickstart.cloudera:2181 --create --topic smoke --partitions 1 --replication-factor 1

created topic smoke

[cloudera@quickstart ~]$ kafka-topics --zookeeper quickstart.cloudera:2181 --listanbu
hello-kafka
kafka-sanity
smoke
test
xx1
[cloudera@quickstart ~]$ kafka-console-producer --broker-list quickstart.cloudera:9092 --topic smoke
>smoke testing for kafka
>checking
>connection

[cloudera@quickstart ~]$ kafka-console-consumer --bootstrap-server quickstart.cloudera:9092 --topic smoke --from-beginning

Could you someone help me what needs to be fixed, why did the consumer not able to  consume those messages.

Zookeeper checks

--------------------------

[cloudera@quickstart ~]$ zookeeper-client
Connecting to localhost:2181
2019-09-24 11:40:48,109 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.5-cdh5.13.0--1, built on 10/04/2017 18:04 GMT
2019-09-24 11:40:48,123 [myid:] - INFO [main:Environment@100] - Client environment:host.name=quickstart.cloudera
2019-09-24 11:40:48,123 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_221
2019-09-24 11:40:48,125 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-09-24 11:40:48,125 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/java/jdk1.8/jre
2019-09-24 11:40:48,125 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/usr/lib/zookeeper/bin/../build/classes:/usr/lib/zookeeper/bin/../build/lib/*.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/usr/lib/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/lib/zookeeper/bin/../lib/jline-2.11.jar:/usr/lib/zookeeper/bin/../zookeeper-3.4.5-cdh5.13.0.jar:/usr/lib/zookeeper/bin/../src/java/lib/*.jar:/etc/zookeeper/conf::/etc/zookeeper/conf:/usr/lib/zookeeper/zookeeper-3.4.5-cdh5.13.0.jar:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/lib/log4j-1.2.16.jar:/usr/lib/zookeeper/lib/netty-3.10.5.Final.jar:/usr/lib/zookeeper/lib/slf4j-log4j12.jar:/usr/lib/zookeeper/lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/lib/jline-2.11.jar
2019-09-24 11:40:48,125 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:os.version=2.6.32-573.el6.x86_64
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:user.name=cloudera
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/home/cloudera
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/home/cloudera
2019-09-24 11:40:48,127 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@306a30c7
Welcome to ZooKeeper!
2019-09-24 11:40:48,151 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2019-09-24 11:40:48,353 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:48990, server: localhost/127.0.0.1:2181
2019-09-24 11:40:48,381 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x16d647f9f970060, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0]

Thanks

Anbu

[#2]

@anbazhagan_muth 

When I look at your scripts, your Kafka scripts should run like shell scripts so your syntax should look like this from the  Kafka install directory

kafka/bin/kafka-console-consumer.sh --bootstrap-server quickstart.cloudera:9092 --topic smoke --from-beginning

Please revert 

[#3]

I'm using Cloudera QuickStart VM 5.13 and I installed their Kafka version.

Listing Kafka Topics:

/usr/bin/kafka-topics --list --zookeeper quickstart.cloudera:2181

Creating Kafka Topic:

/usr/bin/kafka-topics --create --zookeeper quickstart.cloudera:2181 --replication-factor 1 --partitions 3 --topic myFirstTopic

Start a Producer:

/usr/bin/kafka-console-producer --broker-list quickstart.cloudera:9092 --topic myFirstTopic

Start a Consumer:

/usr/bin/kafka-console-consumer --bootstrap-server quickstart.cloudera:9092 --topic myFirstTopic --from-beginning

Notes for your issue:

• Replications need to be less than or equal to the number of brokers.
  • I changed offsets.topic.replication.factor in Kafka configuration from Cloudera Manager and I set it to 1 (Cause I have 1 broker)
•  You can delete brokers from Zookeeper as shown in the link below. Then restart Kafka to recreate these brokers.

Zookeeper CLI:

Accessing Zookeeper CLI:

/usr/bin/zookeeper-client

List All:

ls /

Output:

[cluster, controller, brokers, zookeeper, admin, isr_change_notification, log_dir_event_notification, ngdata, controller_epoch, solr, consumers, latest_producer_id_block, config, hbase]

List Kafka Brokers:

ls /brokers

Output:

[ids, topics, seqid]

List Kafka Topics in Zookeeper:

ls /brokers/topics

Output:

[myFirstTopic, __consumer_offsets]

 Delete a Path in Zookeeper:

rmr /brokers