Support Questions
Find answers, ask questions, and share your expertise

moving data from kafka to hdfs

Solved Go to solution

moving data from kafka to hdfs

New Contributor

 I need a Help i have two question please

1-how I can transform the data from apache kafka to hdfs????

2-how I can transform the data from apache kafka to SparkStreming????

 

thank you

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: moving data from kafka to hdfs

Cloudera Employee

You can use flume or nifi to publish data from kafka to nifi:

 

a. Using flume 

Kafka Source -> Flume -> HDFS

b. Using Nifi:

 

Configure PublishKafka processor --> PutHdfs processor 

 

And to integrate kafka for spark streaming you need to build spark streaming job, refer the below doc. for  more details:

https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_spark-component-guide/content/using-spark-s...

View solution in original post

1 REPLY 1
Highlighted

Re: moving data from kafka to hdfs

Cloudera Employee

You can use flume or nifi to publish data from kafka to nifi:

 

a. Using flume 

Kafka Source -> Flume -> HDFS

b. Using Nifi:

 

Configure PublishKafka processor --> PutHdfs processor 

 

And to integrate kafka for spark streaming you need to build spark streaming job, refer the below doc. for  more details:

https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_spark-component-guide/content/using-spark-s...

View solution in original post