Support Questions
Find answers, ask questions, and share your expertise

Why do the List type processors not accept incoming connections and FlowFiles

New Contributor

Dear community,

I am trying to build a flow where I need to run either a ListFTP or ListSFTP based on some preconditions, furthermore, the ListFTP or ListSFTP's remote path is dynamic, e.g. using the NiFi Expression Language.
The remote path is NOT configured as a NiFi variable, it is gathered during the flow from an external resource.

I took a dive into the source of the standard processors that come with NiFi to learn what is technically happening within those processors and as a consequence I saw that no List type processor accepts incoming connections.
There is probably some good reason for it, but I could not find that reason in neither the source-code nor the developers guide.

I took the liberty of just creating a custom processor extending from ListSFTP as a start:

@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
public class ListDynamiscSFTP extends ListSFTP {
  // ...
}

This in turn allowed me to create incoming connections to it and allowed me to switch between ListFTP or ListSFTP with the use of a RouteOnAttribute processor, albeit with static configuration in each List processor.

Going deeper, the AbstractListProcessor#onTrigger eventually calls AbstractListProcessor#performListing, with no ProcessSession nor FlowFile, removing the ability to call evaluateAttributeExpressions(FlowFile) within the FileTransfer#getListing

Currently I'm using NiFi 1.6.0, but I've also checked out 1.7.0-SNAPSHOT from git which contains no changes to this functionality.

To enable this behavior I need to override a whole bunch of functions, so much so that I question this move by asking this community first, what the reason is behind this before moving on.

So, TL;DR;
What is the reason/logic for not allowing incoming connections and FlowFiles into a List type processor?

Thanks,
Marco

1 ACCEPTED SOLUTION

Super Guru
@Marco Springer

There is similar Hcc thread regarding list processors in NiFi, please refer to this link for more details.

Let us know if you have additional questions..!!

View solution in original post

2 REPLIES 2

Super Guru
@Marco Springer

There is similar Hcc thread regarding list processors in NiFi, please refer to this link for more details.

Let us know if you have additional questions..!!

New Contributor

@Shu

Thank you for that reference!
The 'statefulness of processors' is enough of an explanation.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.