I have a requirement where I need to execute a SQL query to a DB and transform the returned avro file to XML.
First I need to replace a few of the records in the file. So I used a split record processor with Avro reader and XML writer to split the records
Two replace text processors to replace the required texts in each record
Then merge the files into a single file
Once the file is merged, I need to replace the XML declaration and root recrod of XML with custom values.
The resultant file is about 2GB in size. Can I use sed 1,2d command in execute stream command processor to remove the files and replace text processor to prepend the required text.
If I use this method, will the large file occupy system memory and affect performance of other pipelines?