Support Questions
Find answers, ask questions, and share your expertise

Kafka - Can we write kafka message to HDFS

Expert Contributor

Hi Team,

Is there a way to write KAFKA messages to HDFS.

Any kind of suggestions owuld be of great help.

Thank you

2 REPLIES 2

Super Collaborator

@Viswa

Indeed, yes for the archiving purpose ( not suggested for real time).

as the activity is straight forward, you may use flume to do that task with "Kafka source" and "HDFS sink" topology.

documentation can be found here

or you may use NiFi to do the same real time.

Hope that helps!!

Super Collaborator

Yes. This feature exists in many forms

  • Flume
  • MapReduce using Camus or Apache Gobblin
  • Spark Streaming
  • NiFi
  • Streamsets
  • Kafka Connect

Depending on what tools you have available, it's up to you to decide which makes the most sense.

; ;