question Re: transfer file using kafka in Support Questions

transfer file using kafka

Sathish Jeganathan — Wed, 21 Dec 2016 14:57:05 GMT

Hi - i wanna transfer a complete file from one system to other using KAFKA ? can you please help in doing it ?

thanks,

sathish

Re: transfer file using kafka

rajkumar_singh — Wed, 21 Dec 2016 15:01:15 GMT

tailf a file a pipe it to kafka console producer

tailf install.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list `hostname -f`:6667 --topic kafkatopic

or cat a file and pipe it to console producer

cat install.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list `hostname -f`:6667 --topic kafkatopic

Re: transfer file using kafka

Sathish Jeganathan — Thu, 22 Dec 2016 12:47:25 GMT

@Rajkumar Singh

i should have been more specific on my asks... sorry for that... actually i want to ingest/load/insert data from kafka producer to hdfs consumer... is there a way to do it with normal kafka commands or any tools available to do it ?

thanks,

sathish

Re: transfer file using kafka

rajkumar_singh — Thu, 22 Dec 2016 13:04:34 GMT

@sathish jeganathan

for such data ingestion case I will suggest you to use apache nifi, you can use putkafka and puthdfs processor, with the help of these processor you can write your files directly on hdfs.

to refer putHDFS you can follow this document

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.hadoop.PutHDFS/

Re: transfer file using kafka

Sathish Jeganathan — Thu, 22 Dec 2016 13:26:06 GMT

@Rajkumar Singh - how about kafka connect. kafka connect is the official one i believe... how those two (nifi & kafka connect) differs from one another.. ?

thanks,

sathish

Re: transfer file using kafka

rajkumar_singh — Thu, 22 Dec 2016 13:33:23 GMT

@sathish jeganathan kafka connect is good if you have source and destination kafka cluster, this way you can stream the data between both clusters, I have not tried it so can't comment on pros/cons of it but certainly a NiFi has quite rich set of processor with the help of these you can stream the data along with transformation/enrichment of the data

Re: transfer file using kafka

Sathish Jeganathan — Thu, 22 Dec 2016 17:14:29 GMT

@Rajkumar Singh can you please let me know if there is any doc which talks more about kafka ? also i wanted to understand more about data streaming between kafka clusters... and how kafka clusters communicates with each other ?

thanks,

sathish

Re: transfer file using kafka

rajkumar_singh — Thu, 22 Dec 2016 17:17:26 GMT

please follow

https://kafka.apache.org/documentation.html