Read from Kafka, convert data from JSON to CSV and write to HADOOP

[#1]

Hi,

I'm looking for tutorial for the following flow:

1. Read message from Kafka (JSON format)

2. Convert the JSON format to CSV format

3. Write the CSV to Hadoop

It's possible to do it with Nifi?

Thanks.

[#2 · Accepted solution]

@Ya ko

Yes,it's possible with record oriented processors(Convert Record) NiFi,

Flow:

1. Consume Kafka Processor //to read data from Kafka
2. Convert Record processor //to convert json format to csv format
3. PutHDFS //write the csv format data to HDFS

In ConvertRecord processor we are configuring Record Reader(Json Tree Reader) to read incoming Json format data and Record writer Controller service(CSVSetWriter) to write the output flowfile in CSV format, we need to define Avro schema for the Record Reader/Writer controller services.

Follow this tutorial to know more details and configurations regarding Convert record processor.

[#3]

Thank you!