Support Questions

Find answers, ask questions, and share your expertise

Script not working as expected all the time from Execute Stream Command

avatar
Rising Star

I have NiFi 1.23.2 running in cluster mode in K8s. As List SFTP does not take an incoming request have written a script that is called from ExecuteStreamCommand to count the number of files in a directory.
I am using expect to make this happen.

#!/bin/sh -e


USER=$1
export PASSWORD="${SFTP_PASSWORD}"
REMOTE_DIR=$3
HOST=$4

BATCH_FILE=$(mktemp)
echo "spawn sftp $USER@$HOST" > $BATCH_FILE
echo "expect \"*fingerprint*\"" >> $BATCH_FILE
echo "send \"yes\\r\"" >> $BATCH_FILE
echo "expect \"password:\"" >> $BATCH_FILE
echo "send \"\$env(PASSWORD)\\r\"" >> $BATCH_FILE
echo "expect \"sftp>\"" >> $BATCH_FILE
echo "send \"ls -l $REMOTE_DIR \\r\"" >> $BATCH_FILE
echo "expect \"sftp>\"" >> $BATCH_FILE
echo "send \"exit\\r\"" >> $BATCH_FILE

#expect $BATCH_FILE

dirs=$(expect $BATCH_FILE| sed -n '/sftp>/, /sftp>/p' | grep -v '^sftp>'| awk '$0 ~ /^d/ {print $NF}')
rm $BATCH_FILE
total=0

SAVEIFS=$IFS
IFS=$'\n'
directories=($dirs)
IFS=$SAVEIFS


for value in "${directories[@]}"; do
val=$(echo "$value" | tr -d '[:blank:]' | tr -d '[:space:]')
if [[ "$val" == "Completed" || "$val" == "Failed" || "$val" == "Rejected" || "$val" == "Error" ]]; then
dir=$REMOTE_DIR/$val
BATCH_FILE=$(mktemp)
echo "spawn sftp $USER@$HOST" > $BATCH_FILE
echo "expect \"*fingerprint*\"" >> $BATCH_FILE
echo "send \"yes\\r\"" >> $BATCH_FILE
echo "expect \"password:\"" >> $BATCH_FILE
echo "send \"\$env(PASSWORD)\\r\"" >> $BATCH_FILE
echo "expect \"sftp>\"" >> $BATCH_FILE
echo "send \"ls -l $dir\\r\"" >> $BATCH_FILE
echo "expect \"sftp>\"" >> $BATCH_FILE
echo "send \"exit\\r\"" >> $BATCH_FILE
count=$(expect $BATCH_FILE| sed -n '/sftp>/, /sftp>/p' | grep -v '^sftp>'| wc -l)
total=$((total + count))
rm $BATCH_FILE
fi
done

echo "$total"

This script is working fine when I run inside this manually with in the container, also is working fine when NiFi runs on standalone mode in 1.23.0

bash-4.4$ ./test_files.sh
4

test_files.sh has the same script but HOST, USER and REMOTE_DIR hardcoded.

When I call the same script from NiFi processor I am getting

execution.error
send: spawn id exp3 not open while executing "send "yes\r"" (file "/tmp/tmp.eatmyc06B6" line 3)

scoutjohn_0-1694517716576.png

 

The fingerprint expects had to be added inside the container as it is being asked. when running outside NiFi it is not required.

scoutjohn_0-1694520380503.png

 

There are times when it works, in the list. one has an error , the other doesn't

scoutjohn_1-1694520231582.png

 

scoutjohn_0-1694520196406.png

 

I have tried the header with both /bash and /sh.

what could be the reason? Please advise.

1 REPLY 1

avatar
Rising Star

Turns out there is a workaround, there is an option to change the first processor to execute in single node, this will execute flowfiles from single node.

scoutjohn_0-1694527385381.png