Created 06-26-2018 10:52 AM
Dear community,
I am trying to build a flow where I need to run either a ListFTP or ListSFTP based on some preconditions, furthermore, the ListFTP or ListSFTP's remote path is dynamic, e.g. using the NiFi Expression Language.
The remote path is NOT configured as a NiFi variable, it is gathered during the flow from an external resource.
I took a dive into the source of the standard processors that come with NiFi to learn what is technically happening within those processors and as a consequence I saw that no List type processor accepts incoming connections.
There is probably some good reason for it, but I could not find that reason in neither the source-code nor the developers guide.
I took the liberty of just creating a custom processor extending from ListSFTP as a start:
@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED) public class ListDynamiscSFTP extends ListSFTP { // ... }
This in turn allowed me to create incoming connections to it and allowed me to switch between ListFTP or ListSFTP with the use of a RouteOnAttribute processor, albeit with static configuration in each List processor.
Going deeper, the AbstractListProcessor#onTrigger eventually calls AbstractListProcessor#performListing, with no ProcessSession nor FlowFile, removing the ability to call evaluateAttributeExpressions(FlowFile) within the FileTransfer#getListing
Currently I'm using NiFi 1.6.0, but I've also checked out 1.7.0-SNAPSHOT from git which contains no changes to this functionality.
To enable this behavior I need to override a whole bunch of functions, so much so that I question this move by asking this community first, what the reason is behind this before moving on.
So, TL;DR;
What is the reason/logic for not allowing incoming connections and FlowFiles into a List type processor?
Thanks,
Marco
Created 06-26-2018 01:09 PM
There is similar Hcc thread regarding list processors in NiFi, please refer to this link for more details.
Let us know if you have additional questions..!!
Created 06-26-2018 01:09 PM
There is similar Hcc thread regarding list processors in NiFi, please refer to this link for more details.
Let us know if you have additional questions..!!
Created 06-26-2018 01:39 PM
@Shu
Thank you for that reference!
The 'statefulness of processors' is enough of an explanation.
Created 01-12-2024 08:45 AM
Hi @marco21
can you please provide me the full implementation code for ListDynamiscSFTP that you have mentioned in your post. I do have the same requirement but not able to determine the methods that I have to override to get the exact implementation of ListSFTP