HDP Twitter demo - Send data into kafka from csv or text file

[#1]

I am working on setting up Twitter demo in HDP 2.3.2 sandbox and HDP 2.2 sandbox. I need to find a way to load data from csv or text files (twitter feeds are in text file as VPN connection does not allow to access Twitter).

Whats the best way to load data from text file into kafka? I am using this demo

[#2 · Accepted solution]

http://kafka.apache.org/documentation.html#quickstart

Skip down to step #4:

/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list localhost:6667 --topic test < my_input_file.csv

Each line in "my_input_file.csv" will be published to Kafka as an individual message.

[#3]

Hi @Randy Gelhausen, I tried this method, It works fine if you have a kafka topic with single partition, but when I try to ingest messages into a kafka topic with two partitions, only one of the partiton is being hit. Could you suggest a way to ingest messages in multiple partitions.

[#4]

You can alwasy use Nifi. Nifi processor has ability to read a file, split it, drop into kafka.

[#5]

hello friend Neeraj... have u done this task successfully??