Support Questions

Find answers, ask questions, and share your expertise

How to create Kafka topics directly on HDFS



How can we create kafka topics directly into hdfs.


Super Mentor


Do you mean to use some sort of Kafka . HDFS connector?

In that case you might want to refer to :


Or if you would like to check the Apache NiFi using a couple of out-of-the-box processors...

- ConsumeKafka

- MergeContent (to merge together small messages from Kafka to a larger file for HDFS)


For example using ConsumerKafka, you could deliver data from Kafka to HDFS without writing any code, and could make use of NiFi’s MergeContent processor to take messages coming from Kafka and batch them together into appropriately sized files for HDFS.