question HDP Twitter demo - Send data into kafka from csv or text file in Archives of Support Questions (Read Only)

HDP Twitter demo - Send data into kafka from csv or text file

nsabharwal — Wed, 18 Nov 2015 03:10:48 GMT

I am working on setting up Twitter demo in HDP 2.3.2 sandbox and HDP 2.2 sandbox. I need to find a way to load data from csv or text files (twitter feeds are in text file as VPN connection does not allow to access Twitter).

Whats the best way to load data from text file into kafka? I am using this demo

Re: HDP Twitter demo - Send data into kafka from csv or text file

rgelhausen — Wed, 18 Nov 2015 03:12:45 GMT

http://kafka.apache.org/documentation.html#quickstart

Skip down to step #4:

/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list localhost:6667 --topic test < my_input_file.csv

Each line in "my_input_file.csv" will be published to Kafka as an individual message.

Re: HDP Twitter demo - Send data into kafka from csv or text file

hfaouaz — Thu, 19 Nov 2015 02:05:28 GMT

You can alwasy use Nifi. Nifi processor has ability to read a file, split it, drop into kafka.

Re: HDP Twitter demo - Send data into kafka from csv or text file

prks280 — Wed, 23 Nov 2016 01:39:14 GMT

hello friend Neeraj... have u done this task successfully??

Re: HDP Twitter demo - Send data into kafka from csv or text file

ankit_lakhotia — Thu, 09 Feb 2017 21:45:47 GMT

Hi @Randy Gelhausen, I tried this method, It works fine if you have a kafka topic with single partition, but when I try to ingest messages into a kafka topic with two partitions, only one of the partiton is being hit. Could you suggest a way to ingest messages in multiple partitions.