Skip to main content
LakshmiTharun
LakshmiTharun
Silver ⭐️⭐️
October 12, 2023
Question

apache nifi executestreamcommand issue with using rsync

Hi,

 

we recently started evaluating apache nifi for our use cases.

 

Here is one of such sample pipe line that we have constructed

LakshmiTharun_0-1697085857041.png

we have two processed one is Getfile which checks for files present in a local folder. Once they are available it sends them to the execute stream command. 

 

In 

execute stream command we want to send file that we received from getfile step to a remote linux server. The destination linux server doesn't have an FTP server hence cant use putFTP. This is the command I want to execute to transfer the file

sshpass -p '123' rsync -avz -e ssh --ignore-existing "${filename}" test@172.172.172.172:/home/test/nifioutput

 

Here is the configuration for execute stream command

LakshmiTharun_1-1697086635511.pngLakshmiTharun_2-1697086692685.png

 log file

LakshmiTharun_3-1697086803774.png

Any help would be greatly appreciated.

 

 

Regards

Lakshmi Tharun.

 

4 replies

VidyaSargur
Community Manager
VidyaSargur
Community Manager
October 12, 2023

@LakshmiTharun, Welcome to our community! To help you get the best possible answer, I have tagged in our NiFi experts @cotopaul @SAMSAL @MattWho @steven-matison @TimothySpann who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Translate
Regards, Vidya Sargur, Community Manager Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
TimothySpann
Platinum ⭐️⭐️⭐️
TimothySpann
Platinum ⭐️⭐️⭐️
October 12, 2023

put your command in a shell script named yourscript.sh and make sure NiFi has execute permissions.  rsync can be very slow so make sure you have a long timeout on the command.   i have seen rsync take 5 minutes to 20 hours.

Translate
LakshmiTharun
LakshmiTharunAuthor
Silver ⭐️⭐️
October 13, 2023

can we not do it using the way I have done by configuring the properties in executestreamcommand.

Is there a way that I can see the log or else the complete scipt Nifi is trying to execute. This is the complete script that I want to execute

sshpass -p '123' rsync -avz -e ssh --ignore-existing /test.pdf test@172.16.4.59:/home/test/nifioutput


Translate
TimothySpann
Platinum ⭐️⭐️⭐️
TimothySpann
Platinum ⭐️⭐️⭐️
October 13, 2023

it's very hard to format a long call like that without wrapping it.   also you will need to fully list out directories so not /test.pdf but /the/actual/location/even/if/it/s/giant/test.pdf.  it may work.   you may also need to do /usr/bin/sshpass.   it is always easier to just write a simple runme.sh with the text in it, make it executable and test it.    also make sure nifi user has permissions to sshpass and if you are running on a cluster that all machines have that binary and have permissions

Translate
Terms & ConditionsAccessibility statement