question Re: How to get list of files from FetchSFTP or any other way as ListSFTP does not take incoming flowfiles in Support Questions

How to get list of files from FetchSFTP or any other way as ListSFTP does not take incoming flowfiles

scoutjohn — Wed, 23 Aug 2023 14:24:44 GMT

Hi , I am trying to List the files in an SFTP location. Cannot use ListSFTP because listSFTP does not take incoming connection, but the issue for us is the SFTP location is coming from a response of an API. Currently I am using FetchSFTP. but takes only a single input which is a filename, that is incorrect as per the requirements, i need a list of files in a directory.

I came across this ticket [NIFI-4621] Allow inputs to ListSFTP and ListFTP - ASF JIRA (apache.org) which is still in open state.

Is there any alternate way to achieve this? Is there a way to use executeScript and run python sftp ?

Re: How to get list of files from FetchSFTP or any other way as ListSFTP does not take incoming flowfiles

scoutjohn — Thu, 24 Aug 2023 13:34:28 GMT

I think I can achieve this by using ExecuteStreamCommand processor and calling a script.

#!/bin/bash

HOST="host"
USER="testuser"
REMOTE_DIR="/files/data"
export PASSWORD="Qwerty@123"

BATCH_FILE=$(mktemp)
echo "spawn sftp $USER@$HOST" > $BATCH_FILE
echo "expect \"password:\"" >> $BATCH_FILE
echo "send \"\$env(PASSWORD)\\r\"" >> $BATCH_FILE
echo "expect \"sftp>\"" >> $BATCH_FILE
echo "send \"ls -l $REMOTE_DIR\\r\"" >> $BATCH_FILE
echo "expect \"sftp>\"" >> $BATCH_FILE
echo "send \"exit\\r\"" >> $BATCH_FILE

expect $BATCH_FILE | sed -n '/sftp>/, /sftp>/p' | grep -v '^sftp>'| awk '$5 < 1000000000 {print $NF}'

rm $BATCH_FILE

this will give output:
$ ./batchfile.sh
names.txt
names2.txt
numlist.txt
some_witherror.txt

maybe increase the size of the attribute length to include larger value. Of course, for this I will have to install expect in my docker image.

Re: How to get list of files from FetchSFTP or any other way as ListSFTP does not take incoming flowfiles

scoutjohn — Fri, 25 Aug 2023 12:59:25 GMT

this flowfile would be split and send to fetch sftp. Files are picked from there and then deleted after processing