Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

JSON split and then converted to CSV before being merged together again?

avatar
Explorer

Hi, i'm very new to NiFi and i was hoping for some advice. I'm taking in some JSON data which i'm assuming needs to be split before i can successfully convert it to CSV and then merge together again. json.png shows the JSON data before it has been converted, csv.png shows the data after its been converted and merged back together and desiredCSV.png shows what i want the csv file to look like.

I'm currently using SplitJson -> EvaluateJsonPath -> ReplaceText -> MergeContent. Iv'e tried using the ConvertRecord processor but i can't seem to configure it properly.

This is the flow for one record, i then need to do the same thing with over a hundred other records and merge them all together into one big CSV file!?

Any help would be greatly appreciated.

64709-desiredcsv.png

64707-json.png


csv.png
1 ACCEPTED SOLUTION

avatar
Master Guru
@Mark McGowan

You need to change the

Delimiter Strategy

Text

set Demarcator property value to New line(shift + enter)

65396-mergecontent.png

Then you can get the merged file contents in new lines.

Output:-

The merged output flowfile will looks like below with the above properties

2018,1,12,3,0,0
2018,1,12,2,0,0

View solution in original post

5 REPLIES 5

avatar

Your flow looks fine to me. The only thing that you need to take care of it merging ALL the fragments while using MergeContent. So use Defragment merge strategy. This will pick ALL the flow files created from a single document by SplitJson and will make sure that all of them are merged before moving further. Follows a sample MergeContent config.

64711-screen-shot-2018-03-19-at-50056-pm.png

Let know if that works for you or if you need any other help.

PS - If you need any help with your existing flow, refer to this link

avatar
Explorer

Hi Rahul, thanks for your prompt reply. Unfortunately the resulting CSV file is still being merged in the same manner. see following snippet.

64712-csv.png

as you can see i don't have the heading and all csv files are being appended onto the one line.

avatar
Master Guru
@Mark McGowan

You need to change the

Delimiter Strategy

Text

set Demarcator property value to New line(shift + enter)

65396-mergecontent.png

Then you can get the merged file contents in new lines.

Output:-

The merged output flowfile will looks like below with the above properties

2018,1,12,3,0,0
2018,1,12,2,0,0

avatar
Explorer

Thanks Shu, that worked perfect.

avatar
Master Guru
@Mark McGowan

You can do merging by using Correlation attribute name property in merge content processor, to use this property we cannot use convert record processor because each flowfile we need to keep date,hour attrbiutes as one attribute and use the attribute in merge content correlation attribute name. Then the merge content processor will merge with same attribute flowfiles into one flowfile.

Please refer to below links for more details about correlation attribute usage

https://community.hortonworks.com/questions/161827/mergeprocessor-nifi-using-the-correlation-attribu...

https://community.hortonworks.com/questions/55926/using-the-mergecontent-processor-in-nifi-can-i-use...

https://community.hortonworks.com/questions/87178/merge-fileflow-files-based-on-time-rather-than-siz...

For reference Convert record xml

178086-json-to-csv.xml