Created on 09-24-2019 11:41 AM - last edited on 09-26-2019 01:56 PM by ask_bill_brooks
Hello Sir,
After installaing kafka in the cloudera manager i have tested the below code,but the consumer not able to consume those messages.While installing kafka in the cloudera manager,i had given the below two configurations as follows:
kafka-topics --zookeeper quickstart.cloudera:2181 --create --topic smoke --partitions 1 --replication-factor 1
created topic smoke
[cloudera@quickstart ~]$ kafka-topics --zookeeper quickstart.cloudera:2181 --listanbu
hello-kafka
kafka-sanity
smoke
test
xx1
[cloudera@quickstart ~]$ kafka-console-producer --broker-list quickstart.cloudera:9092 --topic smoke
>smoke testing for kafka
>checking
>connection
[cloudera@quickstart ~]$ kafka-console-consumer --bootstrap-server quickstart.cloudera:9092 --topic smoke --from-beginning
Could you someone help me what needs to be fixed, why did the consumer not able to consume those messages.
Zookeeper checks
--------------------------
[cloudera@quickstart ~]$ zookeeper-client
Connecting to localhost:2181
2019-09-24 11:40:48,109 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.5-cdh5.13.0--1, built on 10/04/2017 18:04 GMT
2019-09-24 11:40:48,123 [myid:] - INFO [main:Environment@100] - Client environment:host.name=quickstart.cloudera
2019-09-24 11:40:48,123 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_221
2019-09-24 11:40:48,125 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-09-24 11:40:48,125 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/java/jdk1.8/jre
2019-09-24 11:40:48,125 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/usr/lib/zookeeper/bin/../build/classes:/usr/lib/zookeeper/bin/../build/lib/*.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/usr/lib/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/lib/zookeeper/bin/../lib/jline-2.11.jar:/usr/lib/zookeeper/bin/../zookeeper-3.4.5-cdh5.13.0.jar:/usr/lib/zookeeper/bin/../src/java/lib/*.jar:/etc/zookeeper/conf::/etc/zookeeper/conf:/usr/lib/zookeeper/zookeeper-3.4.5-cdh5.13.0.jar:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/lib/log4j-1.2.16.jar:/usr/lib/zookeeper/lib/netty-3.10.5.Final.jar:/usr/lib/zookeeper/lib/slf4j-log4j12.jar:/usr/lib/zookeeper/lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/lib/jline-2.11.jar
2019-09-24 11:40:48,125 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:os.version=2.6.32-573.el6.x86_64
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:user.name=cloudera
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/home/cloudera
2019-09-24 11:40:48,126 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/home/cloudera
2019-09-24 11:40:48,127 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@306a30c7
Welcome to ZooKeeper!
2019-09-24 11:40:48,151 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2019-09-24 11:40:48,353 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:48990, server: localhost/127.0.0.1:2181
2019-09-24 11:40:48,381 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x16d647f9f970060, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0]
Thanks
Anbu
Created 09-26-2019 12:44 PM
When I look at your scripts, your Kafka scripts should run like shell scripts so your syntax should look like this from the Kafka install directory
kafka/bin/kafka-console-consumer.sh --bootstrap-server quickstart.cloudera:9092 --topic smoke --from-beginning
Please revert
Created 12-06-2020 02:46 PM
I'm using Cloudera QuickStart VM 5.13 and I installed their Kafka version.
Listing Kafka Topics:
/usr/bin/kafka-topics --list --zookeeper quickstart.cloudera:2181
Creating Kafka Topic:
/usr/bin/kafka-topics --create --zookeeper quickstart.cloudera:2181 --replication-factor 1 --partitions 3 --topic myFirstTopic
Start a Producer:
/usr/bin/kafka-console-producer --broker-list quickstart.cloudera:9092 --topic myFirstTopic
Start a Consumer:
/usr/bin/kafka-console-consumer --bootstrap-server quickstart.cloudera:9092 --topic myFirstTopic --from-beginning
Notes for your issue:
Zookeeper CLI:
Accessing Zookeeper CLI:
/usr/bin/zookeeper-client
List All:
ls /
Output:
[cluster, controller, brokers, zookeeper, admin, isr_change_notification, log_dir_event_notification, ngdata, controller_epoch, solr, consumers, latest_producer_id_block, config, hbase]
List Kafka Brokers:
ls /brokers
Output:
[ids, topics, seqid]
List Kafka Topics in Zookeeper:
ls /brokers/topics
Output:
[myFirstTopic, __consumer_offsets]
Delete a Path in Zookeeper:
rmr /brokers