Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Why doesn't ListSFTP allow upstream connections?

avatar

Hello,

I'm working on a flow that currently uses FetchSFTP to download a file from an sFTP server every 15 minutes or so. It works, however, if a file is not there (which is valid in our use-case) then the FetchSFTP throws an error. Because of this error, our log monitors think there is an actual error, but in reality it's not an error, it is just how our flow is set up. So to avoid this error from FetchSFTP, I want to use a ListSFTP to check for a file first. However, ListSFTP does not allow upstream connections so I cannot pass the sFTP server information to it. We load this server information from a GetFile upstream and then use an UpdateAttribute to add the information to the flowfile. I don't understand the reasoning for not allowing upstream connections to the ListSFTP processor. Can someone explain why it's designed like this? Also, do you know of another workaround to stop FetchSFTP from throwing an error when it doesn't find a file (since it's just going to be looking again in 15 minutes or so).

Thanks for the help.

Also, since a picture is worth a thousand words, here is our current flow with the FetchSFTP getting information from the client config file:

60524-flow.png

1 ACCEPTED SOLUTION

avatar
Super Collaborator
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
5 REPLIES 5

avatar

Bump. Does anyone have any ideas surrounding this? I feel like it's a poor design choice to not allow ListSFTP to have upstream connections. It doesn't make sense because almost every field in ListSFTP allows expression language. Why would NiFi not allow us to pass the server information to ListSFTP when it has the capability of expression language? So it seems the only way to pass server information to this processor is via the variable registry in 'nifi.properties'?

avatar
Super Collaborator
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar

Thank you, Ed, for the insight on the ListSFTP processor and for providing some possible workarounds. I think this will give me enough to work with to come up with a solution.

avatar

Hi @eberezitsky  is there a way to extend ListSFTP with the same functionality but that allows inward connection and allows expression language for configuration

avatar

I agree it would be nice to allow incoming upstream connections to ListSFTP. However in answer to the previous comment you can also set variables in a Process Group by right clicking on the canvas and selecting the variable option. It is dynamic and much better for my needs than setting in 'nifi.properties'