Need help on how to make listSFTP processor properties dynamic in NiFi ?

[#1]

Hi Folks,

My Requirement: SFTP connection setup.

Partners send files to us via SFTP. we need to fetch those files and put into respecting folder for further processing.

we are planning to use nifi technology for this sftp setup. Please help to make this happen.

We are thinking below process to implement.

First will use the ListSFTP processor for list all the files and followed by fetchSFTP processor for fetch all the files which are authorized.

Need help on, below points.

1) ListSFTP supports listing of the supplied directory (Remote Path) and supplied "Hostname" using the 'Username" and 'Password" / "Private Key Passphrase".

The password can change at a regular interval (depending on organization policy) or the Hostname or the Remote Path can change based on some other requirement.

in the above case how to set the ListSFTP properties based on the attributes.. like point 2.

( for these attributes, we are planning to prepare the csv file which will have Remorepath ,hostname, usename, and password ,and other attributes...)

2) How to read the csv file and make all make all the attributes as variables...to use in down stream flow.

3) How to route the files into target directories ( if we have sub folders based on partner..how to copy that perticular partner files into other folders.).

Thanks in Advance.

Thanks,
Rangareddy Y

[#2]

ListSFTP is a source processor that does not accept incoming flow files. It is made to track the state of a specific directory on a specific host and find new files, so if those values can change at anytime then the state needs to be reset. This is challenging if those values can change on any given execution while the processor is running, so for this reason the processor does not support incoming flow files and the values can only be changed by stopping and starting the processor with new configuration.

[#3]

Thanks for the reply.

is it possible to do the authentication and dynamic parameter dynamic set in FetchSFTP processor..
if yes, please let me know how to do it..

Thanks,

Rangareddy Y

[#4]

How to read data from multiple sources..if i want to ready 100+ different partner files what is the recommended solution for this.

[#5]

@Bryan Bende

in another post,

http://apache-nifi.1125220.n5.nabble.com/ListSFTP-processor-with-dynamic-hostname-td12495.html

you have mentioned we can achieve this by using executestreamcommand..Can you please suggest me how to prepare sftp connection command for hostname, port, username, password, remotepath....Please share detail information.

or else how to get it thru rest nifi-api calls...can you please give sample nifi processors flow..

Thanks in advance.

Thanks,
Rangareddy Y

[#6]

@Bryan Bende

Can you suggest me how to implement below scenario in nifi.

500+ partners used to send files to us thru SFTP. how to read all these sftp connections for files in nifi. if listSFTP does not support dynamic parameter change.. what is the alternate solution for this.

Can you please give me flow of steps..

Thanks,
Rangareddy Y