Community Articles

Find and share helpful community-sourced technical articles.
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.
Labels (2)
Expert Contributor

In this post, we will learn how to create a Kafka topic and produce and consume messages from a Kafka topic. After testing the basic producer and consumer example, we will test it with Spark using spark-examples.jar file.


Creating a Kafka topic:


# kafka bootstrap server

# kafka topic name

# group name

# creating a topic
/opt/cloudera/parcels/CDH/lib/kafka/bin/ --create --topic ${TOPIC_NAME} --bootstrap-server ${KAFKA_BROKERS}

# describing a topic
/opt/cloudera/parcels/CDH/lib/kafka/bin/ --describe --topic ${TOPIC_NAME} --bootstrap-server ${KAFKA_BROKERS}



Producing messages to Kafka topic:


# producing kafka messages

/opt/cloudera/parcels/CDH/lib/kafka/bin/ --topic ${TOPIC_NAME} --broker-list ${KAFKA_BROKERS}



Consuming messages from Kafka topic:


# consuming kafka messages

/opt/cloudera/parcels/CDH/lib/kafka/bin/ --bootstrap-server ${KAFKA_BROKERS} --group ${GROUP_NAME} --topic ${TOPIC_NAME} --from-beginning



Submitting the Spark KafkaWordCount example:


spark-submit \
--master yarn \
--deploy-mode client \
--packages org.apache.spark:spark-streaming-kafka-0-10_2.11: \
--repositories \
--class org.apache.spark.examples.streaming.DirectKafkaWordCount \
/opt/cloudera/parcels/CDH/lib/spark/examples/jars/spark-examples_*.jar ${KAFKA_BROKERS} ${GROUP_NAME} ${TOPIC_NAME}


Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.
Version history
Last update:
‎04-05-2022 12:46 AM
Updated by:
Top Kudoed Authors