transfer file using kafka

[#1]

Hi - i wanna transfer a complete file from one system to other using KAFKA ? can you please help in doing it ?

thanks,

sathish

[#2 · Accepted solution]

tailf a file a pipe it to kafka console producer

tailf install.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list `hostname -f`:6667 --topic kafkatopic

or cat a file and pipe it to console producer

cat install.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list `hostname -f`:6667 --topic kafkatopic

[#3]

@Rajkumar Singh

i should have been more specific on my asks... sorry for that... actually i want to ingest/load/insert data from kafka producer to hdfs consumer... is there a way to do it with normal kafka commands or any tools available to do it ?

thanks,

sathish

[#4]

@sathish jeganathan

for such data ingestion case I will suggest you to use apache nifi, you can use putkafka and puthdfs processor, with the help of these processor you can write your files directly on hdfs.

to refer putHDFS you can follow this document

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.hadoop.PutHDFS/

[#5]

@Rajkumar Singh - how about kafka connect. kafka connect is the official one i believe... how those two (nifi & kafka connect) differs from one another.. ?

thanks,

sathish

[#6]

@sathish jeganathan kafka connect is good if you have source and destination kafka cluster, this way you can stream the data between both clusters, I have not tried it so can't comment on pros/cons of it but certainly a NiFi has quite rich set of processor with the help of these you can stream the data along with transformation/enrichment of the data

[#7]

@Rajkumar Singh can you please let me know if there is any doc which talks more about kafka ? also i wanted to understand more about data streaming between kafka clusters... and how kafka clusters communicates with each other ?

thanks,

sathish

[#8]

please follow

https://kafka.apache.org/documentation.html