If I am in the process of creating multiple flow files during an ExecuteScript processor, and an error occurs before I'm done, I transfer the original flow file to its failure relationship. My doubt, however, is whether or not I have to do anything with the flow files that I've created thus far. Consider, for example, the following groovy: def flowFile = session.get() if (!flowFile) return def filesDir = flowFile.getAttribute('filesDirectory') // See https://community.hortonworks.com/questions/106878/split-one-nifi-flow-file-into-multiple-flow-file-b.html def flowFiles = [] as List<FlowFile> try { // Get a list of all the trades in the portfolio directory. File folder = new File(filesDir); File[] listOfFiles = folder.listFiles(); for (File file : listOfFiles) { if (file.isFile()) { // Create a new flow file with that single trade. def newFlowFile = session.create(flowFile) flowFiles << newFlowFile // put some attributes here, or do something that might throw an exception } } } catch (e) { session.transfer(flowFile, REL_FAILURE) return } session.remove(flowFile) session.transfer(flowFiles, REL_SUCCESS) My doubt is whether or not I need something like this in the "catch" block: catch (e) { // is this necessary? for (FlowFile theFlowFile : flowFiles) { session.remove(theFlowFile) } session.transfer(flowFile, REL_FAILURE) return } ...or will the session automatically clean them up on the basis that they haven't been directed to any relationship? ... View more

Hello NiFi community, I have been investigating a problem where the most recent data provenance segment in a shut down nifi application session is not available when the nifi application is next started. My investigation has been to repeatedly (20+ times) stop nifi, start nifi (and let it generate a bit of data provenance), and then inspect the data provenance when the UI becomes available to confirm that all previous data provenance segments are accessible. Then repeat, and repeat, and repeat. At first, I was able to reproduce the problem fairly reliably ever second or third time that I stopped nifi and started it again. The data provenance generated during the previous nifi application session was not displayed in the current nifi application session. I could still see the .prov files in the provenance_repository on disk, so I concluded the most recent .prov file from the previous nifi application session had been corrupted. I then discovered this recommendation (https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.1/bk_user-guide/content/bootstrap-conf.html) that suggests that you should comment out UseG1GC in bootstrap.conf when using the write-ahead configuration for provenance. I followed the instructions and commented-out the line in bootstrap.conf. Then, I observed that the data provenance generated during previous sessions of nifi were no longer missing from subsequent nifi sessions. The change appeared to stem whatever problem was occurring. Assuming that it was a corruption problem, disabling UseG1GC seemed to prevent the corruption that I had been observing. This behaviour (to me) seems to confirm that the stability of WriteAheadProvenanceRepository becomes questionable when UseG1GC is configured. I have a few questions related to what I have seen: Does this problem only affect provenance when nifi is being stopped or have other problems been observed for long-running nifi application sessions? If UseG1GC is as much at odds with WriteAheadProvenanceRepository as it seems, why does nifi ship with both configured out of the box? Is there a means by which I can validate the integrity of the .prov files programmatically? Thank you, kindly, for your time and attention. Sean ... View more