How to merge many json files in one csv file in NIFI?

Question

hi, i have the following problem - i am creating myself quite a few json files via AttributeToJSON processor. Here, as destination, I have set flow-file content. Now I would like to combine them into one csv file. I used MergeContent in my flow first to combine them into one json and then ConvertRecord to turn it into csv. However, it does not create one file for me, but many, I do not know if the problem lies in the MergeContent processor or ConvertRecord. Or should I approach it differently? There are my settings in MergeContent

MattWho · Accepted Answer

@MWM

MergeContent is probably not the processor you want to use to merge your JSON files as it will simply concatenate them together instead of creating one larger json. You should use the MergeRecord processor instead. In this processor you can specify a JsonTreeReader to read your source Json files and then use a CSVRecordSetWriter to output merged CSV file(s). The rest of the MergeRecord processor configuration will control how many source Json records are added in to a single output CSV.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.