Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to create Kafka topics directly on HDFS

Highlighted

How to create Kafka topics directly on HDFS

Contributor

Hello,

How can we create kafka topics directly into hdfs.

1 REPLY 1

Re: How to create Kafka topics directly on HDFS

Super Mentor

@Nilesh

Do you mean to use some sort of Kafka . HDFS connector?

In that case you might want to refer to : https://community.hortonworks.com/questions/177417/import-data-from-kafka-topics-to-hdfs.html

.

Or if you would like to check the Apache NiFi using a couple of out-of-the-box processors...

- ConsumeKafka

- MergeContent (to merge together small messages from Kafka to a larger file for HDFS)

- PutHDFS

For example using ConsumerKafka, you could deliver data from Kafka to HDFS without writing any code, and could make use of NiFi’s MergeContent processor to take messages coming from Kafka and batch them together into appropriately sized files for HDFS.

https://bryanbende.com/development/2016/09/15/apache-nifi-and-apache-kafka

.