Support Questions

Find answers, ask questions, and share your expertise

How to create Kafka topics directly on HDFS

Contributor

Hello,

How can we create kafka topics directly into hdfs.

1 REPLY 1

Super Mentor

@Nilesh

Do you mean to use some sort of Kafka . HDFS connector?

In that case you might want to refer to : https://community.hortonworks.com/questions/177417/import-data-from-kafka-topics-to-hdfs.html

.

Or if you would like to check the Apache NiFi using a couple of out-of-the-box processors...

- ConsumeKafka

- MergeContent (to merge together small messages from Kafka to a larger file for HDFS)

- PutHDFS

For example using ConsumerKafka, you could deliver data from Kafka to HDFS without writing any code, and could make use of NiFi’s MergeContent processor to take messages coming from Kafka and batch them together into appropriately sized files for HDFS.

https://bryanbende.com/development/2016/09/15/apache-nifi-and-apache-kafka

.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.