Creating multiple flow files and ExecuteScript error handling

[#1]

If I am in the process of creating multiple flow files during an ExecuteScript processor, and an error occurs before I'm done, I transfer the original flow file to its failure relationship. My doubt, however, is whether or not I have to do anything with the flow files that I've created thus far.

Consider, for example, the following groovy:

def flowFile = session.get()

if (!flowFile) return

def filesDir = flowFile.getAttribute('filesDirectory')

// See https://community.hortonworks.com/questions/106878/split-one-nifi-flow-file-into-multiple-flow-file-...

def flowFiles = [] as List<FlowFile>

try {

// Get a list of all the trades in the portfolio directory.

File folder = new File(filesDir);

File[] listOfFiles = folder.listFiles();

for (File file : listOfFiles) {

if (file.isFile()) {

// Create a new flow file with that single trade.

def newFlowFile = session.create(flowFile)

flowFiles << newFlowFile

// put some attributes here, or do something that might throw an exception

}

}

}

catch (e) {

session.transfer(flowFile, REL_FAILURE)

return

}

session.remove(flowFile)

session.transfer(flowFiles, REL_SUCCESS)

My doubt is whether or not I need something like this in the "catch" block:

catch (e) {

// is this necessary?

for (FlowFile theFlowFile : flowFiles) {

session.remove(theFlowFile)

}

session.transfer(flowFile, REL_FAILURE)

return

}

...or will the session automatically clean them up on the basis that they haven't been directed to any relationship?

[#2 · Accepted solution]

Once a flow file has been created in a session, it must be removed or transferred before the session is committed (which happens at the end of ExecuteScript). Since your try is outside the loop that creates new flow files, you'll want to remove all the created ones, namely the flowFiles list. You can do that with simply:

session.remove(flowFiles)

rather than the loop you have in your catch statement.

[#3]

Thanks, Matt.

[#4]

@Matt Burgess

Maybe there is a spot somewhere this may be mentioned in your already-excellent ExecuteScript Cookbook articles for the next person?

[#5]

That's a great idea, thanks! I've been meaning to update it, hopefully sooner than later 🙂