Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Creating multiple flow files and ExecuteScript error handling

avatar

If I am in the process of creating multiple flow files during an ExecuteScript processor, and an error occurs before I'm done, I transfer the original flow file to its failure relationship. My doubt, however, is whether or not I have to do anything with the flow files that I've created thus far.

Consider, for example, the following groovy:


def flowFile = session.get()

if (!flowFile) return


def filesDir = flowFile.getAttribute('filesDirectory')


// See https://community.hortonworks.com/questions/106878/split-one-nifi-flow-file-into-multiple-flow-file-...

def flowFiles = [] as List<FlowFile>


try {

// Get a list of all the trades in the portfolio directory.

File folder = new File(filesDir);

File[] listOfFiles = folder.listFiles();

for (File file : listOfFiles) {

if (file.isFile()) {

// Create a new flow file with that single trade.


def newFlowFile = session.create(flowFile)

flowFiles << newFlowFile

// put some attributes here, or do something that might throw an exception

}

}

}

catch (e) {

session.transfer(flowFile, REL_FAILURE)

return

}


session.remove(flowFile)

session.transfer(flowFiles, REL_SUCCESS)




My doubt is whether or not I need something like this in the "catch" block:


catch (e) {

// is this necessary?

for (FlowFile theFlowFile : flowFiles) {

session.remove(theFlowFile)

}


session.transfer(flowFile, REL_FAILURE)

return

}


...or will the session automatically clean them up on the basis that they haven't been directed to any relationship?


1 ACCEPTED SOLUTION

avatar
Master Guru

Once a flow file has been created in a session, it must be removed or transferred before the session is committed (which happens at the end of ExecuteScript). Since your try is outside the loop that creates new flow files, you'll want to remove all the created ones, namely the flowFiles list. You can do that with simply:

session.remove(flowFiles)

rather than the loop you have in your catch statement.

View solution in original post

4 REPLIES 4

avatar
Master Guru

Once a flow file has been created in a session, it must be removed or transferred before the session is committed (which happens at the end of ExecuteScript). Since your try is outside the loop that creates new flow files, you'll want to remove all the created ones, namely the flowFiles list. You can do that with simply:

session.remove(flowFiles)

rather than the loop you have in your catch statement.

avatar

Thanks, Matt.

avatar

@Matt Burgess

Maybe there is a spot somewhere this may be mentioned in your already-excellent ExecuteScript Cookbook articles for the next person?

avatar
Master Guru

That's a great idea, thanks! I've been meaning to update it, hopefully sooner than later 🙂