Can you please suggest me, on below scenario. We are planning to implement this scenario in nifi
listSFTP processor need to read data from multiple sources. each one has their own hostname, port, username, password, and remotepath.
I am thinking to use nifi-api to change the properties of processors dynamically. LOOP CONCEPT
Can you please suggest me how to make nifi-api loop to call the listSFTP processor and each time that need to be changed..
we know all the attribute values...these values should update into listSFTP processor thru nifi rest api..
Can you please suggest me how to do it..
Thanks in advance,
Is there a reason you have decided to use the NiFi API to perform this task? You could read from a configuration the list of SFTP endpoints (along with username, password, etc.), parse that, and store those values as FlowFile attributes that are passed into the ListSFTP processor. You can re-use the ListSFTP processor that way, much like you are envisioning.
NiFi has the NiFi expression language which can be used to reference FlowFile attributes inside of processor configuration values.
yes. reason is there.. we have 500+ partners sending files to us. we have to read..each one has diff hostname,port, username, and password. It will be difficult for us to use 500+ listSFTP processors for each connection..
i do not know how to change the listSFTP processor proeprties dynamically..that is the reason i choose rest api call..
Please suggest me, is there any other alternate way we have..
Upon further research, I see what you're running into. The processor does not accept incoming connections so you cannot give it any dynamic properties. Unfortunately because of the way that processor keeps track of state and keeps track of which files it has already retrieved, it does not support dynamically changing the properties of its source.
See this question from 2016 on the same topic:
You can try the suggestions outlined there. There is also the option of extending the processor to do what you want to do.