Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Read from Kafka, convert data from JSON to CSV and write to HADOOP

Solved Go to solution
Highlighted

Read from Kafka, convert data from JSON to CSV and write to HADOOP

Explorer

Hi,

I'm looking for tutorial for the following flow:

1. Read message from Kafka (JSON format)

2. Convert the JSON format to CSV format

3. Write the CSV to Hadoop

It's possible to do it with Nifi?

Thanks.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Read from Kafka, convert data from JSON to CSV and write to HADOOP

Super Guru

@Ya ko

Yes,it's possible with record oriented processors(Convert Record) NiFi,

Flow:

1. Consume Kafka Processor //to read data from Kafka
2. Convert Record processor //to convert json format to csv format
3. PutHDFS //write the csv format data to HDFS

In ConvertRecord processor we are configuring Record Reader(Json Tree Reader) to read incoming Json format data and Record writer Controller service(CSVSetWriter) to write the output flowfile in CSV format, we need to define Avro schema for the Record Reader/Writer controller services.

Follow this tutorial to know more details and configurations regarding Convert record processor.

View solution in original post

2 REPLIES 2
Highlighted

Re: Read from Kafka, convert data from JSON to CSV and write to HADOOP

Super Guru

@Ya ko

Yes,it's possible with record oriented processors(Convert Record) NiFi,

Flow:

1. Consume Kafka Processor //to read data from Kafka
2. Convert Record processor //to convert json format to csv format
3. PutHDFS //write the csv format data to HDFS

In ConvertRecord processor we are configuring Record Reader(Json Tree Reader) to read incoming Json format data and Record writer Controller service(CSVSetWriter) to write the output flowfile in CSV format, we need to define Avro schema for the Record Reader/Writer controller services.

Follow this tutorial to know more details and configurations regarding Convert record processor.

View solution in original post

Re: Read from Kafka, convert data from JSON to CSV and write to HADOOP

Explorer

Thank you!

Don't have an account?
Coming from Hortonworks? Activate your account here