Support Questions

Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Read from Kafka, convert data from JSON to CSV and write to HADOOP

Explorer

Hi,

I'm looking for tutorial for the following flow:

1. Read message from Kafka (JSON format)

2. Convert the JSON format to CSV format

3. Write the CSV to Hadoop

It's possible to do it with Nifi?

Thanks.

1 ACCEPTED SOLUTION

Super Guru

@Ya ko

Yes,it's possible with record oriented processors(Convert Record) NiFi,

Flow:

1. Consume Kafka Processor //to read data from Kafka
2. Convert Record processor //to convert json format to csv format
3. PutHDFS //write the csv format data to HDFS

In ConvertRecord processor we are configuring Record Reader(Json Tree Reader) to read incoming Json format data and Record writer Controller service(CSVSetWriter) to write the output flowfile in CSV format, we need to define Avro schema for the Record Reader/Writer controller services.

Follow this tutorial to know more details and configurations regarding Convert record processor.

View solution in original post

2 REPLIES 2

Super Guru

@Ya ko

Yes,it's possible with record oriented processors(Convert Record) NiFi,

Flow:

1. Consume Kafka Processor //to read data from Kafka
2. Convert Record processor //to convert json format to csv format
3. PutHDFS //write the csv format data to HDFS

In ConvertRecord processor we are configuring Record Reader(Json Tree Reader) to read incoming Json format data and Record writer Controller service(CSVSetWriter) to write the output flowfile in CSV format, we need to define Avro schema for the Record Reader/Writer controller services.

Follow this tutorial to know more details and configurations regarding Convert record processor.

Explorer

Thank you!

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.