Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Testing out Apache Kafka

Testing out Apache Kafka

Explorer

I am currently testing on Apache Kafka, followed an example I found and managed to pump some data into Kafka.

My question now is how can I retrieve the messages in kafka and store it in hdfs?

 

Any advise/tutorials/examples that can share with me?

 

Thanks.

1 REPLY 1

Re: Testing out Apache Kafka

Super Collaborator
Here is a blog post that gives a good walkthrough of using flume to pull messages from kafka and deliver to hdfs: http://blog.cloudera.com/blog/2014/11/flafka-apache-flume-meets-apache-kafka-for-event-processing/

And here is the cloudera documentation detailing the options for configuring flume to work with kafka:
http://www.cloudera.com/content/cloudera/en/documentation/cloudera-kafka/latest/topics/kafka_flume.h...
Don't have an account?
Coming from Hortonworks? Activate your account here