Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to reuse FTP connection properties?

Highlighted

How to reuse FTP connection properties?

Explorer

I'm developing a flow where ListFTP is used to discover new files at a remote FTP server. Connection data (host, port, username and password) is entered in the properties for ListFTP. Downstream I do some processing to determine which files to fetch and where to place them locally (and some logging). Fetching is done with FetchFTP.

The question is how to reuse the FTP connection from ListFTP upstream? I have tried to reuse the properties still attached to flowfiles from the upstream ListFTP, but FetchFTP is giving me some trouble using them. One issue is a NumberFormatException when reusing property for port, both as ${ftp.remote.port} and also when trying ${ftp.remote.port:toNumber()}. Another issue is how to reuse the password from ListFTP upstream.

The flow is supposed to be wrapped in a template, since several remote FTP servers should be setup like this, reusing fetch logic, but having different connection data. So I'd like to only having to enter these one time for each template use, and have connection then reused as needed downstream.

Suggestions on how best to accomplish this?

2 REPLIES 2
Highlighted

Re: How to reuse FTP connection properties?

Apache NiFi 1.4.0 introduced variables per process group (right click anywhere on canvas and select "Variables" to see the variables for current group).

If you wrap your example flow in a process group and made everything reference a variable, then each time you import your template you would just go into the new process group and set the variables accordingly.

Highlighted

Re: How to reuse FTP connection properties?

Explorer

Thanks, Bryan, for bringing this to my attention. Unfortunately I currently only have access to version 1.2 and even just 1.1 through the version of HDF I have at hand. Look forward to a version of HDF with this feature.

I wonder already whether these process group variables will have support for securely managing passwords as part of variable sets, or it will just be plain values.

Don't have an account?
Coming from Hortonworks? Activate your account here