Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

KAFKA CONSUMER NOT CONSUMING MESSAGES - Cloudera quickstart VM

KAFKA CONSUMER NOT CONSUMING MESSAGES - Cloudera quickstart VM

Hello Sir,

 

After installaing kafka in the cloudera manager i have tested the below code,but the consumer not able to consume those messages.While installing kafka in the cloudera manager,i had given the below two configurations as follows:

 

Destination Broker List ===> quickstart.cloudera:9092
bootstrap.servers
 
Source Broker List===> quickstart.cloudera:9092
source.bootstrap.servers
 
  • Java Heap Size of Broker (broker_max_heap_size) =“256”
  • Advertised Host (advertised.host.name) = “quickstart.cloudera”
  • Inter Broker Protocol = “PLAINTEXT”

 

kafka-topics --zookeeper quickstart.cloudera:2181 --create --topic smoke --partitions 1 --replication-factor 1

created topic smoke

 

[cloudera@quickstart ~]$ kafka-topics --zookeeper quickstart.cloudera:2181 --listanbu
hello-kafka
kafka-sanity
smoke
test
xx1
[cloudera@quickstart ~]$ kafka-console-producer --broker-list quickstart.cloudera:9092 --topic smoke
>smoke testing for kafka
>checking
>connection

 

[cloudera@quickstart ~]$ kafka-console-consumer --bootstrap-server quickstart.cloudera:9092 --topic smoke --from-beginning

 

Could you someone help me what needs to be fixed, why did the consumer not able to  consume those messages.

 

Zookeeper checks

--------------------------

 

[cloudera@quickstart ~]$ zookeeper-client
Connecting to localhost:2181
2019-09-24 11:40:48,109 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.5-cdh5.13.0--1, built on 10/04/2017 18:04 GMT
2019-09-24 11:40:48,123 [myid:] - INFO [main:Environment@100] - Client environment:host.name=quickstart.cloudera
2019-09-24 11:40:48,123 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_221
2019-09-24 11:40:48,125 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-09-24 11:40:48,125 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/java/jdk1.8/jre
2019-09-24 11:40:48,125 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/usr/lib/zookeeper/bin/../build/classes:/usr/lib/zookeeper/bin/../build/lib/*.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/usr/lib/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/lib/zookeeper/bin/../lib/jline-2.11.jar:/usr/lib/zookeeper/bin/../zookeeper-3.4.5-cdh5.13.0.jar:/usr/lib/zookeeper/bin/../src/java/lib/*.jar:/etc/zookeeper/conf::/etc/zookeeper/conf:/usr/lib/zookeeper/zookeeper-3.4.5-cdh5.13.0.jar:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/lib/log4j-1.2.16.jar:/usr/lib/zookeeper/lib/netty-3.10.5.Final.jar:/usr/lib/zookeeper/lib/slf4j-log4j12.jar:/usr/lib/zookeeper/lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/lib/jline-2.11.jar
2019-09-24 11:40:48,125 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:os.version=2.6.32-573.el6.x86_64
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:user.name=cloudera
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/home/cloudera
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/home/cloudera
2019-09-24 11:40:48,127 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@306a30c7
Welcome to ZooKeeper!
2019-09-24 11:40:48,151 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2019-09-24 11:40:48,353 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:48990, server: localhost/127.0.0.1:2181
2019-09-24 11:40:48,381 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x16d647f9f970060, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0]

 

 

Thanks

Anbu

 

 

1 REPLY 1
Highlighted

Re: KAFKA CONSUMER NOT CONSUMING MESSAGES - Cloudera quickstart VM

Mentor

@anbazhagan_muth 

When I look at your scripts, your Kafka scripts should run like shell scripts so your syntax should look like this from the  Kafka install directory

kafka/bin/kafka-console-consumer.sh --bootstrap-server quickstart.cloudera:9092 --topic smoke --from-beginning

Please revert