Support Questions
Find answers, ask questions, and share your expertise

How to fetch files form a SFTP location with name appended with todays date, facing problem with the regular expression when trying with GetSFTP processor in NIFI

Re: How to fetch files form a SFTP location with name appended with todays date, facing problem with the regular expression when trying with GetSFTP processor in NIFI

So , to get this working..I need to listSFTP?, if yes..how do i pass it as parameter to fetch the files

Re: How to fetch files form a SFTP location with name appended with todays date, facing problem with the regular expression when trying with GetSFTP processor in NIFI

Master Guru

@anoop_u_shet 
@Ash1 

Let me try to answer both question here since they are related.

1. Can you set multiple java regular expressions in on single ListSFTP processor?  No
If you can come up with one single java regex, that can do your filtering, that will work.

2. Can you have one ListSFTP processor connect to two SFTP servers running on two different target servers? No.  You would need to have one ListSFTP processor for each target SFTP server.

3. Can I have just one FetchSFTP processor with multiple ListSFTP processors feeding it?  Yes ***
The ListSFTP processor is designed to create FlowFile with numerous FlowFile attributes/metadata and no content.  That metadata is then used by the FetchSFTP processor to retrieve tat FlowFile's content. 
*** If each target SFTP server uses passwords to authenticate, you may need to use different fetchSFTP processors since passwords can not be stored in FlowFile attributes.

Take a look at the ListSFTP [1] processor documentation/usage (available within yoru install directly from UI under help or on Apache NiFi site) and look at the list of Attributes it writes to the NiFi FlowFiles it creates.  You'll see below as some of those attributes:
sftp.remote.host
sftp.remote.port
filename
path

In the FetchSFTP [2] processor, these Attributes from the FlowFile should be used to fetch that FlowFile's content using NiFi Expression Language (NEL) [3].  NEL is commonly used through out your dataflow(s). It allows utilize attributes from FlowFiles (among other things) in processors.

For example, your FetchSFTP may look like this:

MattWho_0-1623847696059.png

You'll see 4 places where I used NEL.  NEL expressions always start with "${" and end with "}".  Above examples are very simple in that they only read the attribute name form the source FlowFile and use the value assigned to that FlowFile Attribute in the processor configuration property.


NOTE: Not all processor properties support NEL.  Floating your cursor over the "?" icon in a property will tell you via popup if NEL is supported by that property.

[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apach...

[2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apach...

[3] https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html

If my responses assisted with yoru queries, please take a moment to click "Accept" on all responses that contributed to a solution for you.

Thank you,
Matt

 

Re: How to fetch files form a SFTP location with name appended with todays date, facing problem with the regular expression when trying with GetSFTP processor in NIFI

New Contributor

Thanks you very much Matt .. this definitely helped a lot .