Execute commands on remote SFTP server from NiFi

[#1]

Hello,

I have my NiFi application running on 1 server. And I need to connect to users SFTP server and need to execute few commands like cd, rm, rename etc on remote server.

And to connect to that server I need to fulfill 2 step authentication (username/password and key authentication)

So if i ssh and provide key from shell script which will be mounted in docker. For that i will have to add the key needs to be installed in all nodes and then needs to be added to users sftp server.

It does not seem to be best solution

I there any other way where  I can use NiFi processor to execute these commands to remote sftp server which needs to be connected through 2 step authentication

Any advise will be very helpful

Thank you.

Shalaka

[#2]

@Elsaa ,

If you can create a shell script that does what you need, you can use NiFi's ExecuteProcess to execute that shell script.

André

[#3]

@araujo 

If I use shell script I will have to establish 2 step authentication connection from shell script to execute the commands to remote server.

In that case I will have to add public key from all possible nodes (where my docker image will be installed) to remote sftp server. I believe it is not a good solution.

To avoid this I wanted to use NiFi processor.

2 possible scenarioes-

1. If I can use NiFi processor to execute commands on remote sftp server directly

2. If I can create keys and install it to node through docker image (so even if docker is intalled on another node the key will remain same which is added to remote server). In this case i can just use shell script which will establish connection and execute commands.

[#4]

I'm not sure if it's possible run SFTP commands other than put/get in NiFi.

Sorry,

André

[#5]

Hello Andre,

Thank you for the response. Yes I couldnt find a way where we can run SFTP commands to remote server in processor.

Any idea if we can put file as "filename1.part" and then after file is put then rename to "filename2.xml" at remote SFTP server in NiFi processor?

Thanks,

Shalaka

[#6 · Accepted solution]

---

---

Hi @Elsaa ,

It's possible to do something similar by setting "Dot Rename = true" in the SFTP processor.

If that's set, while your file is being uploaded NiFi will create the file with a dot prefix(e.g.  .filename1.xml) and when the upload finishes it will rename the file to its original name.

Cheers,

André

[#7]

I want it to pass as filename.part  and rename to filename.xml

I think I see one more option as temporary rename which renames file as mentioned name while transfer and then rename to original once file is transferred.

[#8]

That's true! That option fits perfectly to your needs.

André