Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Convert JSON to CSV in nifi

Highlighted

Convert JSON to CSV in nifi

I want to convert JSON files to CSV in nifi. We can achieve this in Python and other programming languages and have multiple articles on it. I have multiple JSON files and each file has different schema(one specific file will have one schema only). I can see there are templates to convert CSV to JSON and other conversions. But I didn't see any template to convert JSON data to CSV. I have gone through the article https://community.hortonworks.com/articles/64069/converting-a-large-json-file-into-csv.html and https://community.hortonworks.com/questions/63995/convert-json-to-csv-using-nifi.html ,however here we are hard coding the schema. As I have multiple files and each file has different schema, I can't hardcode the schema. Any suggestions please.

2 REPLIES 2
Highlighted

Re: Convert JSON to CSV in nifi

Hi @User 805,

You could use the InferAvroSchema processor to infer the schema from your JSON data and put this schema as an attribute of the flow files, then you can define your JSON Reader controller service to use this attribute as the schema source.

Note that not defining in a schema registry the schema of each source you have might source of errors: if some of your JSONs are missing some fields, you will miss columns in the generated CSV.

Also infering the schema for every single flow file can be costly in terms of performance.

What I'd recommend is to use the InferAvroSchema processor to help you generating the schema for each one of your sources and then put the schemas in the schema registry of your choice and use it. I'll be more performant and you can leverage the features of the schema registry such as schema versioning, backward/forward compatibility, reuse of the schemas over multiple components, etc.

Hope this helps.

Highlighted

Re: Convert JSON to CSV in nifi

Don't have an account?
Coming from Hortonworks? Activate your account here