question Re: JSON split and then converted to CSV before being merged together again? in Archives of Support Questions (Read Only)

JSON split and then converted to CSV before being merged together again?

mcgmark87 — Sun, 18 Aug 2019 07:46:30 GMT

Hi, i'm very new to NiFi and i was hoping for some advice. I'm taking in some JSON data which i'm assuming needs to be split before i can successfully convert it to CSV and then merge together again. json.png shows the JSON data before it has been converted, csv.png shows the data after its been converted and merged back together and desiredCSV.png shows what i want the csv file to look like.

I'm currently using SplitJson -> EvaluateJsonPath -> ReplaceText -> MergeContent. Iv'e tried using the ConvertRecord processor but i can't seem to configure it properly.

This is the flow for one record, i then need to do the same thing with over a hundred other records and merge them all together into one big CSV file!?

Any help would be greatly appreciated.

Re: JSON split and then converted to CSV before being merged together again?

RahulSoni — Sun, 18 Aug 2019 07:46:17 GMT

Your flow looks fine to me. The only thing that you need to take care of it merging ALL the fragments while using MergeContent. So use Defragment merge strategy. This will pick ALL the flow files created from a single document by SplitJson and will make sure that all of them are merged before moving further. Follows a sample MergeContent config.

Let know if that works for you or if you need any other help.

PS - If you need any help with your existing flow, refer to this link

Re: JSON split and then converted to CSV before being merged together again?

mcgmark87 — Sun, 18 Aug 2019 07:46:09 GMT

Hi Rahul, thanks for your prompt reply. Unfortunately the resulting CSV file is still being merged in the same manner. see following snippet.

as you can see i don't have the heading and all csv files are being appended onto the one line.

Re: JSON split and then converted to CSV before being merged together again?

Shu_ashu — Sun, 18 Aug 2019 07:46:01 GMT

@Mark McGowan

You need to change the

Delimiter Strategy

Text

set Demarcator property value to New line(shift + enter)

Then you can get the merged file contents in new lines.

Output:-

The merged output flowfile will looks like below with the above properties

2018,1,12,3,0,0
2018,1,12,2,0,0

Re: JSON split and then converted to CSV before being merged together again?

mcgmark87 — Tue, 20 Mar 2018 05:54:56 GMT

Thanks Shu, that worked perfect.

Re: JSON split and then converted to CSV before being merged together again?

Shu_ashu — Sat, 24 Mar 2018 23:50:16 GMT

@Mark McGowan

You can do merging by using Correlation attribute name property in merge content processor, to use this property we cannot use convert record processor because each flowfile we need to keep date,hour attrbiutes as one attribute and use the attribute in merge content correlation attribute name. Then the merge content processor will merge with same attribute flowfiles into one flowfile.

Please refer to below links for more details about correlation attribute usage

https://community.hortonworks.com/questions/161827/mergeprocessor-nifi-using-the-correlation-attribut.html

https://community.hortonworks.com/questions/55926/using-the-mergecontent-processor-in-nifi-can-i-use.html

https://community.hortonworks.com/questions/87178/merge-fileflow-files-based-on-time-rather-than-siz.html

For reference Convert record xml

178086-json-to-csv.xml