Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to get list of files from FetchSFTP or any other way as ListSFTP does not take incoming flowfiles

avatar
Contributor

Hi , I am trying to List the files in an SFTP location. Cannot use ListSFTP because listSFTP does not take incoming connection, but the issue for us is the SFTP location is coming from a response of an API. Currently I am using FetchSFTP. but takes only a single input which is a filename, that is incorrect as per the requirements, i need a list of files in a directory.

I came across this ticket [NIFI-4621] Allow inputs to ListSFTP and ListFTP - ASF JIRA (apache.org) which is still in open state.

 

Is there any alternate way to achieve this? Is there a way to use executeScript and run python sftp ?

2 REPLIES 2

avatar
Contributor

I think I can achieve this by using ExecuteStreamCommand processor and calling a script. 

#!/bin/bash

HOST="host"
USER="testuser"
REMOTE_DIR="/files/data"
export PASSWORD="Qwerty@123"


BATCH_FILE=$(mktemp)
echo "spawn sftp $USER@$HOST" > $BATCH_FILE
echo "expect \"password:\"" >> $BATCH_FILE
echo "send \"\$env(PASSWORD)\\r\"" >> $BATCH_FILE
echo "expect \"sftp>\"" >> $BATCH_FILE
echo "send \"ls -l $REMOTE_DIR\\r\"" >> $BATCH_FILE
echo "expect \"sftp>\"" >> $BATCH_FILE
echo "send \"exit\\r\"" >> $BATCH_FILE


expect $BATCH_FILE | sed -n '/sftp>/, /sftp>/p' | grep -v '^sftp>'| awk '$5 < 1000000000 {print $NF}'

rm $BATCH_FILE

this will give output
$ ./batchfile.sh
names.txt
names2.txt
numlist.txt
some_witherror.txt

maybe increase the size of the attribute length to include larger value. Of course, for this I will have to install expect in my docker image.

avatar
Contributor

scoutjohn_0-1692968150270.png

this flowfile would be split and send to fetch sftp. Files are picked from there and then deleted after processing