Cloudera Kafka

[#1]

I am trying to run Kafka 1.0.1 on a single-node cdh5.13.0. 

I was able to create a topic by: 

Kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 -partitions 1 --topic kafka_demo5

But i have issues when running producer and consumer:

Kafka-console-producer --broker-list quickstart.cloudera:9092 --topic kafka_demo5

Kafka-console-consumer --bootstrap-server quickstart.cloudera:9092 --topic kafka_demo5 --from-beginning

When I run producer-consumer commands I get a similar but very long output [pic]. Output also doesn't list any error if occurred.

 

[#2]

Hi @HarpreetSingh31 

It's not clear to me what the issue is when you say you have problems running the producer and consumer. The outputs you're seeing are normal and it looks to be functioning as expected. 

I'm going to assume here that the problem you're referring to is that you can't seem to be reading the messages you're sending to the topic. Based on the information you posted, where you highlighted that you're only running on Kafka broker, I believe the problem here is that you need to go and change the Kafka configuration offsets.topic.replication.factor and be sure to set that to 1. In Cloudera Manager we have always set that to 3 by default and I had filed an improvement internally about this to ensure that we do not set this to three when a user installs a new Kafka service with brokers less than that number. 

If you look at the Kafka broker log you'll see an error like the one below or something similar:

Number of alive brokers '1' does not meet the required replication factor '3' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet.

You can make the change from:

Cloudera Manager > Kafka > Configuration > Search for 'offsets.topic.replication.factor'

After you change this value you will need to restart your Kafka service. 

Be aware that once you set this to 1, your internal __consumer_offsets topic (used by consumers to commit their offsets) will be created with a single replication factor and this won't change even as you add more brokers to your cluster. If in the future you need to add more brokers then you will have to expand the replication factor for this topic using the kafka-reassign-partition tool:
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools

Hope this helps!