Support Questions
Find answers, ask questions, and share your expertise

JSON split and then converted to CSV before being merged together again?

Solved Go to solution

JSON split and then converted to CSV before being merged together again?

Explorer

Hi, i'm very new to NiFi and i was hoping for some advice. I'm taking in some JSON data which i'm assuming needs to be split before i can successfully convert it to CSV and then merge together again. json.png shows the JSON data before it has been converted, csv.png shows the data after its been converted and merged back together and desiredCSV.png shows what i want the csv file to look like.

I'm currently using SplitJson -> EvaluateJsonPath -> ReplaceText -> MergeContent. Iv'e tried using the ConvertRecord processor but i can't seem to configure it properly.

This is the flow for one record, i then need to do the same thing with over a hundred other records and merge them all together into one big CSV file!?

Any help would be greatly appreciated.

64709-desiredcsv.png

64707-json.png


csv.png
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: JSON split and then converted to CSV before being merged together again?

Super Guru
@Mark McGowan

You need to change the

Delimiter Strategy

Text

set Demarcator property value to New line(shift + enter)

65396-mergecontent.png

Then you can get the merged file contents in new lines.

Output:-

The merged output flowfile will looks like below with the above properties

2018,1,12,3,0,0
2018,1,12,2,0,0

View solution in original post

5 REPLIES 5
Highlighted

Re: JSON split and then converted to CSV before being merged together again?

Your flow looks fine to me. The only thing that you need to take care of it merging ALL the fragments while using MergeContent. So use Defragment merge strategy. This will pick ALL the flow files created from a single document by SplitJson and will make sure that all of them are merged before moving further. Follows a sample MergeContent config.

64711-screen-shot-2018-03-19-at-50056-pm.png

Let know if that works for you or if you need any other help.

PS - If you need any help with your existing flow, refer to this link

Highlighted

Re: JSON split and then converted to CSV before being merged together again?

Explorer

Hi Rahul, thanks for your prompt reply. Unfortunately the resulting CSV file is still being merged in the same manner. see following snippet.

64712-csv.png

as you can see i don't have the heading and all csv files are being appended onto the one line.

Highlighted

Re: JSON split and then converted to CSV before being merged together again?

Super Guru
@Mark McGowan

You need to change the

Delimiter Strategy

Text

set Demarcator property value to New line(shift + enter)

65396-mergecontent.png

Then you can get the merged file contents in new lines.

Output:-

The merged output flowfile will looks like below with the above properties

2018,1,12,3,0,0
2018,1,12,2,0,0

View solution in original post

Highlighted

Re: JSON split and then converted to CSV before being merged together again?

Explorer

Thanks Shu, that worked perfect.

Highlighted

Re: JSON split and then converted to CSV before being merged together again?

Super Guru
@Mark McGowan

You can do merging by using Correlation attribute name property in merge content processor, to use this property we cannot use convert record processor because each flowfile we need to keep date,hour attrbiutes as one attribute and use the attribute in merge content correlation attribute name. Then the merge content processor will merge with same attribute flowfiles into one flowfile.

Please refer to below links for more details about correlation attribute usage

https://community.hortonworks.com/questions/161827/mergeprocessor-nifi-using-the-correlation-attribu...

https://community.hortonworks.com/questions/55926/using-the-mergecontent-processor-in-nifi-can-i-use...

https://community.hortonworks.com/questions/87178/merge-fileflow-files-based-on-time-rather-than-siz...

For reference Convert record xml

178086-json-to-csv.xml

Don't have an account?