@acasta, Don't get me wrong, but I highly doubt that NiFi is ignoring/deleting somehow the files, without your intervention or configurations. What I would suggest you are the following two actions: - First of all, add an LogMessage/LogAttribute after your have unzipped all those files. Basically, double your success queue from your processors where you unzip your tar file and log each file which was extracted. In this way, you get a list with all the files extracted out of your zip file. Make sure to set the Queue as single node to check nifi-app.logs on a single node. - Next, add another LogMessage/LogAttribute after your processor with which you save the data into your Bucket. Send the name of the unzipped files into the logs to get a list with all the files which have saved into your bucket.Make sure to set the Queue as single node to check nifi-app.logs on a single node. Afterwards, you can compare the lists and see if you have extracted and saved all your files. If the lists are 1:1, it means that the problem is not related to NiFi itself, but to something else: like another system doing something in your bucket, having files with the same name which get over written, etc. Another option would be to use DEBUG on all your processors and use RUN ONCE until you process everything you have to process and analyze in real time what is happening.
... View more
