I have a requirement where I need to execute a SQL query to a DB and transform the returned avro file to XML. First I need to replace a few of the records in the file. So I used a split record processor with Avro reader and XML writer to split the records Two replace text processors to replace the required texts in each record Then merge the files into a single file Once the file is merged, I need to replace the XML declaration and root recrod of XML with custom values.  The resultant file is about 2GB in size. Can I use sed 1,2d command in execute stream command processor to remove the files and replace text processor to prepend the required text.  If I use this method, will the large file occupy system memory and affect performance of other pipelines? Flow is attached   ... View more

I have a setup where there is 1 S3 bucket for each day and 4k files per bucket. The requirement I have is  1) Compress all the files 2) archive the files 3) push the files to different bucket 4) delete the source files after the success of the above three operations.    Please suggest a nifi flow for this requirement.      What I have tried:   I tried listing the files using ListS3, Fetching the files using FetchS3, Compress content and tar them using merge content processor. After merge I could not get the filenames to delete from source bucket.    ... View more

I need to monitor the flowfiles that are queued for more than a day in NiFi. Is there a way to monitor that automatically? I have about 1400 processors running in production. It is very tedious to monitor it manually.  ... View more