Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kafka - Can we write kafka message to HDFS

Highlighted

Kafka - Can we write kafka message to HDFS

Expert Contributor

Hi Team,

Is there a way to write KAFKA messages to HDFS.

Any kind of suggestions owuld be of great help.

Thank you

2 REPLIES 2

Re: Kafka - Can we write kafka message to HDFS

Super Collaborator

@Viswa

Indeed, yes for the archiving purpose ( not suggested for real time).

as the activity is straight forward, you may use flume to do that task with "Kafka source" and "HDFS sink" topology.

documentation can be found here

or you may use NiFi to do the same real time.

Hope that helps!!

Re: Kafka - Can we write kafka message to HDFS

Super Collaborator

Yes. This feature exists in many forms

  • Flume
  • MapReduce using Camus or Apache Gobblin
  • Spark Streaming
  • NiFi
  • Streamsets
  • Kafka Connect

Depending on what tools you have available, it's up to you to decide which makes the most sense.

Don't have an account?
Coming from Hortonworks? Activate your account here