Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Creating multiple flow files and ExecuteScript error handling

Solved Go to solution

Creating multiple flow files and ExecuteScript error handling

New Contributor

If I am in the process of creating multiple flow files during an ExecuteScript processor, and an error occurs before I'm done, I transfer the original flow file to its failure relationship. My doubt, however, is whether or not I have to do anything with the flow files that I've created thus far.

Consider, for example, the following groovy:


def flowFile = session.get()

if (!flowFile) return


def filesDir = flowFile.getAttribute('filesDirectory')


// See https://community.hortonworks.com/questions/106878/split-one-nifi-flow-file-into-multiple-flow-file-...

def flowFiles = [] as List<FlowFile>


try {

// Get a list of all the trades in the portfolio directory.

File folder = new File(filesDir);

File[] listOfFiles = folder.listFiles();

for (File file : listOfFiles) {

if (file.isFile()) {

// Create a new flow file with that single trade.


def newFlowFile = session.create(flowFile)

flowFiles << newFlowFile

// put some attributes here, or do something that might throw an exception

}

}

}

catch (e) {

session.transfer(flowFile, REL_FAILURE)

return

}


session.remove(flowFile)

session.transfer(flowFiles, REL_SUCCESS)




My doubt is whether or not I need something like this in the "catch" block:


catch (e) {

// is this necessary?

for (FlowFile theFlowFile : flowFiles) {

session.remove(theFlowFile)

}


session.transfer(flowFile, REL_FAILURE)

return

}


...or will the session automatically clean them up on the basis that they haven't been directed to any relationship?


1 ACCEPTED SOLUTION

Accepted Solutions

Re: Creating multiple flow files and ExecuteScript error handling

Once a flow file has been created in a session, it must be removed or transferred before the session is committed (which happens at the end of ExecuteScript). Since your try is outside the loop that creates new flow files, you'll want to remove all the created ones, namely the flowFiles list. You can do that with simply:

session.remove(flowFiles)

rather than the loop you have in your catch statement.

4 REPLIES 4

Re: Creating multiple flow files and ExecuteScript error handling

Once a flow file has been created in a session, it must be removed or transferred before the session is committed (which happens at the end of ExecuteScript). Since your try is outside the loop that creates new flow files, you'll want to remove all the created ones, namely the flowFiles list. You can do that with simply:

session.remove(flowFiles)

rather than the loop you have in your catch statement.

Re: Creating multiple flow files and ExecuteScript error handling

New Contributor

Thanks, Matt.

Re: Creating multiple flow files and ExecuteScript error handling

New Contributor

@Matt Burgess

Maybe there is a spot somewhere this may be mentioned in your already-excellent ExecuteScript Cookbook articles for the next person?

Re: Creating multiple flow files and ExecuteScript error handling

That's a great idea, thanks! I've been meaning to update it, hopefully sooner than later :)

Don't have an account?
Coming from Hortonworks? Activate your account here