Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ways to get data from Kafka to HDFS

Ways to get data from Kafka to HDFS

Explorer

I'm looking for ways to get data from Kafka to Python.

Currently I'm using this pipeline. Has anyone faced issues with using Flume?

 Flume(exec-source and Kafka-sink) --> Kafka --> Flume(kafka-source and HDFS-sink)

Other options: In case I have a kafka-consumer written, is there a python way of getting the data from Kafka consumer to HDFS (other than Confluent's Connect API)?

Or are there any other means I can get the data from Kafka t HDFS?

5 REPLIES 5
Highlighted

Re: Ways to get data from Kafka to HDFS

Expert Contributor

Hi Swaapnika, I've tried using Flume for that and had no issues.

Investigate this repository for python https://github.com/edenhill/librdkafka. This is the most exhaustive one I guess.

Highlighted

Re: Ways to get data from Kafka to HDFS

Explorer

I see Flume is deprecated and will be removed from HDP in the future releases as mentioned in the HDP-2.6.2-Release Notes. Are there any other techniques that could be used with kafka to get data into HDFS?

Highlighted

Re: Ways to get data from Kafka to HDFS

Super Collaborator

@Swaapnika Guntaka You could use Spark Streaming in PySpark to consume a topic and write the data to HDFS.

You could also use HDF with NiFi and skip Python entirely.

Also, this is a Python client, by Confluent, not related to Kafka Connect. https://github.com/confluentinc/confluent-kafka-python

Highlighted

Re: Ways to get data from Kafka to HDFS

Explorer

Is there a difference between the kafka-connector in the python module and the confluent's one? This is the gihub link for the one mentioned in the python module,

Highlighted

Re: Ways to get data from Kafka to HDFS

Super Collaborator

Confluent is the support company for Kafka. I personally would trust their code more than someone else's.

Don't have an account?
Coming from Hortonworks? Activate your account here