Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

moving data from kafka to hdfs

Solved Go to solution

moving data from kafka to hdfs

New Contributor

 I need a Help i have two question please

1-how I can transform the data from apache kafka to hdfs????

2-how I can transform the data from apache kafka to SparkStreming????

 

thank you

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: moving data from kafka to hdfs

Cloudera Employee

You can use flume or nifi to publish data from kafka to nifi:

 

a. Using flume 

Kafka Source -> Flume -> HDFS

b. Using Nifi:

 

Configure PublishKafka processor --> PutHdfs processor 

 

And to integrate kafka for spark streaming you need to build spark streaming job, refer the below doc. for  more details:

https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_spark-component-guide/content/using-spark-s...

View solution in original post

1 REPLY 1
Highlighted

Re: moving data from kafka to hdfs

Cloudera Employee

You can use flume or nifi to publish data from kafka to nifi:

 

a. Using flume 

Kafka Source -> Flume -> HDFS

b. Using Nifi:

 

Configure PublishKafka processor --> PutHdfs processor 

 

And to integrate kafka for spark streaming you need to build spark streaming job, refer the below doc. for  more details:

https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_spark-component-guide/content/using-spark-s...

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here