Member since
06-26-2018
2
Posts
0
Kudos Received
0
Solutions
06-26-2018
01:39 PM
@Shu Thank you for that reference! The 'statefulness of processors' is enough of an explanation.
... View more
06-26-2018
10:52 AM
Dear community, I am trying to build a flow where I need to run either a ListFTP or ListSFTP based on some preconditions, furthermore, the ListFTP or ListSFTP's remote path is dynamic, e.g. using the NiFi Expression Language. The remote path is NOT configured as a NiFi variable, it is gathered during the flow from an external resource. I took a dive into the source of the standard processors that come with NiFi to learn what is technically happening within those processors and as a consequence I saw that no List type processor accepts incoming connections. There is probably some good reason for it, but I could not find that reason in neither the source-code nor the developers guide. I took the liberty of just creating a custom processor extending from ListSFTP as a start: @InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
public class ListDynamiscSFTP extends ListSFTP {
// ...
} This in turn allowed me to create incoming connections to it and allowed me to switch between ListFTP or ListSFTP with the use of a RouteOnAttribute processor, albeit with static configuration in each List processor. Going deeper, the AbstractListProcessor#onTrigger eventually calls AbstractListProcessor#performListing, with no ProcessSession nor FlowFile, removing the ability to call evaluateAttributeExpressions(FlowFile) within the FileTransfer#getListing Currently I'm using NiFi 1.6.0, but I've also checked out 1.7.0-SNAPSHOT from git which contains no changes to this functionality. To enable this behavior I need to override a whole bunch of functions, so much so that I question this move by asking this community first, what the reason is behind this before moving on. So, TL;DR; What is the reason/logic for not allowing incoming connections and FlowFiles into a List type processor? Thanks, Marco
... View more
Labels:
- Labels:
-
Apache NiFi