Running DirectKafkaWordCount example in CDP

[#1]

In this post, we will learn how to create a Kafka topic and produce and consume messages from a Kafka topic. After testing the basic producer and consumer example, we will test it with Spark using spark-examples.jar file.

Creating a Kafka topic:

# kafka bootstrap server
KAFKA_BROKERS="localhost:9092"

# kafka topic name
TOPIC_NAME="word_count_topic"

# group name
GROUP_NAME="spark-kafka-group"

# creating a topic
/opt/cloudera/parcels/CDH/lib/kafka/bin/kafka-topics.sh --create --topic ${TOPIC_NAME} --bootstrap-server ${KAFKA_BROKERS}

# describing a topic
/opt/cloudera/parcels/CDH/lib/kafka/bin/kafka-topics.sh --describe --topic ${TOPIC_NAME} --bootstrap-server ${KAFKA_BROKERS}

Producing messages to Kafka topic:

# producing kafka messages

/opt/cloudera/parcels/CDH/lib/kafka/bin/kafka-console-producer.sh --topic ${TOPIC_NAME} --broker-list ${KAFKA_BROKERS}

Consuming messages from Kafka topic:

# consuming kafka messages

/opt/cloudera/parcels/CDH/lib/kafka/bin/kafka-console-consumer.sh --bootstrap-server ${KAFKA_BROKERS} --group ${GROUP_NAME} --topic ${TOPIC_NAME} --from-beginning

Submitting the Spark KafkaWordCount example:

spark-submit \
--master yarn \
--deploy-mode client \
--packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.4.7.7.1.7.0-551 \
--repositories https://repository.cloudera.com/artifactory/cloudera-repos/ \
--class org.apache.spark.examples.streaming.DirectKafkaWordCount \
/opt/cloudera/parcels/CDH/lib/spark/examples/jars/spark-examples_*.jar ${KAFKA_BROKERS} ${GROUP_NAME} ${TOPIC_NAME}