Bronze ⭐️⭐️⭐️

Solved

moving data from kafka to hdfs

1 reply
2697 views
Translate All

I need a Help i have two question please

1-how I can transform the data from apache kafka to hdfs????

2-how I can transform the data from apache kafka to SparkStreming????

thank you

Gold ⭐️

You can use flume or nifi to publish data from kafka to nifi:

a. Using flume

Kafka Source -> Flume -> HDFS

b. Using Nifi:

Configure PublishKafka processor --> PutHdfs processor

And to integrate kafka for spark streaming you need to build spark streaming job, refer the below doc. for more details:

https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_spark-component-guide/content/using-spark-streaming.html

Gold ⭐️

You can use flume or nifi to publish data from kafka to nifi:

a. Using flume

Kafka Source -> Flume -> HDFS

b. Using Nifi:

Configure PublishKafka processor --> PutHdfs processor

And to integrate kafka for spark streaming you need to build spark streaming job, refer the below doc. for more details:

https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_spark-component-guide/content/using-spark-streaming.html

Translate