- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
How to merge many json files in one csv file in NIFI?
- Labels:
-
Apache NiFi
Created ‎08-03-2023 06:56 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
hi, i have the following problem - i am creating myself quite a few json files via AttributeToJSON processor. Here, as destination, I have set flow-file content. Now I would like to combine them into one csv file. I used MergeContent in my flow first to combine them into one json and then ConvertRecord to turn it into csv. However, it does not create one file for me, but many, I do not know if the problem lies in the MergeContent processor or ConvertRecord. Or should I approach it differently?
There are my settings in MergeContent
Created ‎08-04-2023 01:37 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@MWM
MergeContent is probably not the processor you want to use to merge your JSON files as it will simply concatenate them together instead of creating one larger json. You should use the MergeRecord processor instead. In this processor you can specify a JsonTreeReader to read your source Json files and then use a CSVRecordSetWriter to output merged CSV file(s). The rest of the MergeRecord processor configuration will control how many source Json records are added in to a single output CSV.
If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.
Thank you,
Matt
Created ‎08-04-2023 01:37 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@MWM
MergeContent is probably not the processor you want to use to merge your JSON files as it will simply concatenate them together instead of creating one larger json. You should use the MergeRecord processor instead. In this processor you can specify a JsonTreeReader to read your source Json files and then use a CSVRecordSetWriter to output merged CSV file(s). The rest of the MergeRecord processor configuration will control how many source Json records are added in to a single output CSV.
If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.
Thank you,
Matt
