Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Execute commands on remote SFTP server from NiFi

avatar
Explorer

Hello,

 

I have my NiFi application running on 1 server. And I need to connect to users SFTP server and need to execute few commands like cd, rm, rename etc on remote server.

 

And to connect to that server I need to fulfill 2 step authentication (username/password and key authentication)

 

So if i ssh and provide key from shell script which will be mounted in docker. For that i will have to add the key needs to be installed in all nodes and then needs to be added to users sftp server.

 

It does not seem to be best solution

 

I there any other way where  I can use NiFi processor to execute these commands to remote sftp server which needs to be connected through 2 step authentication

 

Any advise will be very helpful

 

Thank you.

Shalaka

1 ACCEPTED SOLUTION

avatar
Super Guru


Hi @Elsaa ,

 

It's possible to do something similar by setting "Dot Rename = true" in the SFTP processor.

If that's set, while your file is being uploaded NiFi will create the file with a dot prefix(e.g.  .filename1.xml) and when the upload finishes it will rename the file to its original name.

 

araujo_0-1646119065498.png

 

Cheers,

André

 

 

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

View solution in original post

7 REPLIES 7

avatar
Super Guru

@Elsaa ,

 

If you can create a shell script that does what you need, you can use NiFi's ExecuteProcess to execute that shell script.

 

André

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar
Explorer

 

 

 

 

@araujo 

If I use shell script I will have to establish 2 step authentication connection from shell script to execute the commands to remote server.

In that case I will have to add public key from all possible nodes (where my docker image will be installed) to remote sftp server. I believe it is not a good solution.

To avoid this I wanted to use NiFi processor.

 

2 possible scenarioes-

1. If I can use NiFi processor to execute commands on remote sftp server directly

2. If I can create keys and install it to node through docker image (so even if docker is intalled on another node the key will remain same which is added to remote server). In this case i can just use shell script which will establish connection and execute commands.

avatar
Super Guru

I'm not sure if it's possible run SFTP commands other than put/get in NiFi.

 

Sorry,

André

 

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar
Explorer

Hello Andre,

Thank you for the response. Yes I couldnt find a way where we can run SFTP commands to remote server in processor.

Any idea if we can put file as "filename1.part" and then after file is put then rename to "filename2.xml" at remote SFTP server in NiFi processor?

 

Thanks,

Shalaka

avatar
Super Guru


Hi @Elsaa ,

 

It's possible to do something similar by setting "Dot Rename = true" in the SFTP processor.

If that's set, while your file is being uploaded NiFi will create the file with a dot prefix(e.g.  .filename1.xml) and when the upload finishes it will rename the file to its original name.

 

araujo_0-1646119065498.png

 

Cheers,

André

 

 

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar
Explorer

I want it to pass as filename.part  and rename to filename.xml

I think I see one more option as temporary rename which renames file as mentioned name while transfer and then rename to original once file is transferred.

 

avatar
Super Guru

That's true! That option fits perfectly to your needs.

 

André

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.