Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HDP Twitter demo - Send data into kafka from csv or text file

avatar
Master Mentor

I am working on setting up Twitter demo in HDP 2.3.2 sandbox and HDP 2.2 sandbox. I need to find a way to load data from csv or text files (twitter feeds are in text file as VPN connection does not allow to access Twitter).

Whats the best way to load data from text file into kafka? I am using this demo

1 ACCEPTED SOLUTION

avatar

http://kafka.apache.org/documentation.html#quickstart

Skip down to step #4:

/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list localhost:6667 --topic test < my_input_file.csv

Each line in "my_input_file.csv" will be published to Kafka as an individual message.

View solution in original post

4 REPLIES 4

avatar

http://kafka.apache.org/documentation.html#quickstart

Skip down to step #4:

/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list localhost:6667 --topic test < my_input_file.csv

Each line in "my_input_file.csv" will be published to Kafka as an individual message.

avatar
New Contributor

Hi @Randy Gelhausen, I tried this method, It works fine if you have a kafka topic with single partition, but when I try to ingest messages into a kafka topic with two partitions, only one of the partiton is being hit. Could you suggest a way to ingest messages in multiple partitions.

avatar
Rising Star

You can alwasy use Nifi. Nifi processor has ability to read a file, split it, drop into kafka.

avatar
New Contributor

hello friend Neeraj... have u done this task successfully??