NiFi ListSFTP variable issue

[#1]

Hello guys,

i recently started using NiFi 1.5.0, i want to transfer some files from sftp to S3.

I want to get the variables necessary for ListSFTP ( like ssh_key, username, ecc) from a DB fetching with executeSQL processor. I want to override the processor group variables with executeScript, do you have any ideas? Or is there a way to pass variables to ListSFTP dynamically?

Thanks in advice

[#2 · Accepted solution]

@Semih Gurbuz

-

Since the ListSFTP processor does not accept incoming connection, NiFi Expression Language (EL) variables cannot be provided dynamically. This properties that support EL expect to get those variable values from one of the following:

1. Process group variables (can change these manually or through rest-api calls)

2. NiFi registry-file (defined in nifi.properties. Edits to this file require NiFi restart)

3. NiFi JVM properties (defined in nifi bootstrap.conf. Edits to this file require NiFi restart)

4. NiFI service user environment variables (Edits to NiFi service user env variables requires Nifi restart to be read)

-

This leaves you with Process group variables as the only viable possible solution here.

Here is an example curl commands that can be used to create or update a key/value variable on an existing process group. This example assumes the NiFi has been secured and authorization is via LDAP.

-

# Obtain a user authentication token:

token=$(curl -k 'https://<hostname>:<port>/nifi-api/access/token'  --data 'username=<username>&password=<password>’)

-

#Get current process group variable-registry details:

curl -k 'https://<hostname>:<port>/nifi-api/process-groups/<process group uuid>/variable-registry' -H "Authorization: Bearer $token"

-

#create or update a variable on a specific process group:

curl -k 'https://<hostname>:<port>/nifi-api/process-groups/<process group uuid>/variable-registry/update-requests' -H "Authorization: Bearer $token" -H 'Content-Type: application/json' --data-binary '{"processGroupRevision":{"clientId":"<client id from above response>","version":2},"variableRegistry":{"processGroupId":"<process group uuid>","variables":[{"variable":{"name":"listsftp-server","value":"Server-xyz"}}]}}' --compressed

-

Note: If you Target Nifi is unsecured you will not need to worry about authentication or authorization. So above command would not require "-k" or "-H "Authorization: Bearer $token". The URL in that case would also be http instead of https.

-

So you should be able to retrieve your values from the DB and use a series of invokeHTTP processors to edit the Process group variables. The challenge comes when you need to make sure the listSFTP has executed using the new variables before making another update to the process group variables.

-

Hope this gets you started in the right direction.

Thank you,

Matt

-

If you found this Answer addressed your original question, please take a moment to login and click "Accept" below the answer.