Why do the List type processors not accept incoming connections and FlowFiles

[#1]

Dear community,

I am trying to build a flow where I need to run either a ListFTP or ListSFTP based on some preconditions, furthermore, the ListFTP or ListSFTP's remote path is dynamic, e.g. using the NiFi Expression Language.
The remote path is NOT configured as a NiFi variable, it is gathered during the flow from an external resource.

I took a dive into the source of the standard processors that come with NiFi to learn what is technically happening within those processors and as a consequence I saw that no List type processor accepts incoming connections.
There is probably some good reason for it, but I could not find that reason in neither the source-code nor the developers guide.

I took the liberty of just creating a custom processor extending from ListSFTP as a start:

@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
public class ListDynamiscSFTP extends ListSFTP {
  // ...
}

This in turn allowed me to create incoming connections to it and allowed me to switch between ListFTP or ListSFTP with the use of a RouteOnAttribute processor, albeit with static configuration in each List processor.

Going deeper, the AbstractListProcessor#onTrigger eventually calls AbstractListProcessor#performListing, with no ProcessSession nor FlowFile, removing the ability to call evaluateAttributeExpressions(FlowFile) within the FileTransfer#getListing

Currently I'm using NiFi 1.6.0, but I've also checked out 1.7.0-SNAPSHOT from git which contains no changes to this functionality.

To enable this behavior I need to override a whole bunch of functions, so much so that I question this move by asking this community first, what the reason is behind this before moving on.

So, TL;DR;
What is the reason/logic for not allowing incoming connections and FlowFiles into a List type processor?

Thanks,
Marco

[#2 · Accepted solution]

@Marco Springer

There is similar Hcc thread regarding list processors in NiFi, please refer to this link for more details.

Let us know if you have additional questions..!!

[#3]

@Shu

Thank you for that reference!
The 'statefulness of processors' is enough of an explanation.

[#4]

Hi @marco21 

can you please provide me the full implementation code for ListDynamiscSFTP that you have mentioned in your post. I do have the same requirement but not able to determine the methods that I have to override to get the exact implementation of ListSFTP