How to get list of files from FetchSFTP or any other way as ListSFTP does not take incoming flowfiles

[#1]

Hi , I am trying to List the files in an SFTP location. Cannot use ListSFTP because listSFTP does not take incoming connection, but the issue for us is the SFTP location is coming from a response of an API. Currently I am using FetchSFTP. but takes only a single input which is a filename, that is incorrect as per the requirements, i need a list of files in a directory.

I came across this ticket [NIFI-4621] Allow inputs to ListSFTP and ListFTP - ASF JIRA (apache.org) which is still in open state.

Is there any alternate way to achieve this? Is there a way to use executeScript and run python sftp ?

[#2]

I think I can achieve this by using ExecuteStreamCommand processor and calling a script. 

#!/bin/bash

HOST="host"
USER="testuser"
REMOTE_DIR="/files/data"
export PASSWORD="Qwerty@123"

BATCH_FILE=$(mktemp)
echo "spawn sftp $USER@$HOST" > $BATCH_FILE
echo "expect \"password:\"" >> $BATCH_FILE
echo "send \"\$env(PASSWORD)\\r\"" >> $BATCH_FILE
echo "expect \"sftp>\"" >> $BATCH_FILE
echo "send \"ls -l $REMOTE_DIR\\r\"" >> $BATCH_FILE
echo "expect \"sftp>\"" >> $BATCH_FILE
echo "send \"exit\\r\"" >> $BATCH_FILE

expect $BATCH_FILE | sed -n '/sftp>/, /sftp>/p' | grep -v '^sftp>'| awk '$5 < 1000000000 {print $NF}'

rm $BATCH_FILE

this will give output: 
$ ./batchfile.sh
names.txt
names2.txt
numlist.txt
some_witherror.txt

maybe increase the size of the attribute length to include larger value. Of course, for this I will have to install expect in my docker image.

[#3]

this flowfile would be split and send to fetch sftp. Files are picked from there and then deleted after processing