I have NiFi 1.23.2 running in cluster mode in K8s. As List SFTP does not take an incoming request have written a script that is called from ExecuteStreamCommand to count the number of files in a directory.
I am using expect to make this happen.
#!/bin/sh -e
USER=$1
export PASSWORD="${SFTP_PASSWORD}"
REMOTE_DIR=$3
HOST=$4
BATCH_FILE=$(mktemp)
echo "spawn sftp $USER@$HOST" > $BATCH_FILE
echo "expect \"*fingerprint*\"" >> $BATCH_FILE
echo "send \"yes\\r\"" >> $BATCH_FILE
echo "expect \"password:\"" >> $BATCH_FILE
echo "send \"\$env(PASSWORD)\\r\"" >> $BATCH_FILE
echo "expect \"sftp>\"" >> $BATCH_FILE
echo "send \"ls -l $REMOTE_DIR \\r\"" >> $BATCH_FILE
echo "expect \"sftp>\"" >> $BATCH_FILE
echo "send \"exit\\r\"" >> $BATCH_FILE
#expect $BATCH_FILE
dirs=$(expect $BATCH_FILE| sed -n '/sftp>/, /sftp>/p' | grep -v '^sftp>'| awk '$0 ~ /^d/ {print $NF}')
rm $BATCH_FILE
total=0
SAVEIFS=$IFS
IFS=$'\n'
directories=($dirs)
IFS=$SAVEIFS
for value in "${directories[@]}"; do
val=$(echo "$value" | tr -d '[:blank:]' | tr -d '[:space:]')
if [[ "$val" == "Completed" || "$val" == "Failed" || "$val" == "Rejected" || "$val" == "Error" ]]; then
dir=$REMOTE_DIR/$val
BATCH_FILE=$(mktemp)
echo "spawn sftp $USER@$HOST" > $BATCH_FILE
echo "expect \"*fingerprint*\"" >> $BATCH_FILE
echo "send \"yes\\r\"" >> $BATCH_FILE
echo "expect \"password:\"" >> $BATCH_FILE
echo "send \"\$env(PASSWORD)\\r\"" >> $BATCH_FILE
echo "expect \"sftp>\"" >> $BATCH_FILE
echo "send \"ls -l $dir\\r\"" >> $BATCH_FILE
echo "expect \"sftp>\"" >> $BATCH_FILE
echo "send \"exit\\r\"" >> $BATCH_FILE
count=$(expect $BATCH_FILE| sed -n '/sftp>/, /sftp>/p' | grep -v '^sftp>'| wc -l)
total=$((total + count))
rm $BATCH_FILE
fi
done
echo "$total"
This script is working fine when I run inside this manually with in the container, also is working fine when NiFi runs on standalone mode in 1.23.0
bash-4.4$ ./test_files.sh
4
test_files.sh has the same script but HOST, USER and REMOTE_DIR hardcoded.
When I call the same script from NiFi processor I am getting
execution.error
send: spawn id exp3 not open while executing "send "yes\r"" (file "/tmp/tmp.eatmyc06B6" line 3)
The fingerprint expects had to be added inside the container as it is being asked. when running outside NiFi it is not required.
There are times when it works, in the list. one has an error , the other doesn't
I have tried the header with both /bash and /sh.
what could be the reason? Please advise.
Turns out there is a workaround, there is an option to change the first processor to execute in single node, this will execute flowfiles from single node.
