NiFi SFTP Input missing

[#1]

Hey guys,

i've been a quite reader in this forums last months and you all provide me a lot of hints and solutions for multiple questions.

Now i've search the documentation and this community, but i'm the first one with this problem i think.

We store all our remote-server ips in a database, the login is the same for all. We want to fetch files (names changes possible) from remote servers and download them.

The only (built in) sftp processor with input connection is "GetSFTP", but i have to know the filename to perform a download. i cannot guarantee that the filenames will never change, so this is no way to solve my problem ...

I know, i can create an processor on my own with some development, but my first step is to get it work with built-in ones.

Does anybody got a good idea how to solve it or have i to create a processor on my own?

Greetings from Dortmund

[#2]

@mbraunerde Before GetSFTP, you need this processor first:

ListSFTP 

ListSFTP will send GetSFTP the the list of files to get.  Be sure to search in your NiFi Processor window, SFTP, to see all matching processors.  This works for any keyword, and I use it often.

Docs here:  https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.16.0/org.apach...

[#3]

Hey @steven-matison ,

thanks for replying, but that’s my problem - getsftp provides files and listsftp the data to get it, but listsftp doesn’t allow incoming connections to provide the option per server based on a sql query.

listsftp doc says

Input requirement:

This component does not allow an incoming relationship.

and that’s the point - I need a way to provide filename to fetch or ip to connect 🤷🏼‍:male_sign:

[#4]

Ahh that certainly is a different challenge which would require a slightly different approach.  My best recommendation, outside of making your own processors,  would be a combination of api calls to prepare variables for a processor group that uses listsftp and fetchsftp or getsftp once the variables are setup.   

[#5]

Hey steven-matison,

Thank you for the reply, but I think that way is more complicated to get it supported over multiple versions and easy to understand for my next-year-me.

[#6 · Accepted solution]

I've created a simple processor on my own => https://github.com/mbraunerDE/nifi-plugins/blob/4cd76e047af88e2ec84f51c883b40f74d1bb9ef0/nifi-mbraun...