Skip to main content
Elsaa
Elsaa
Silver ⭐️⭐️
February 23, 2022
Solved

Execute commands on remote SFTP server from NiFi

Hello,

 

I have my NiFi application running on 1 server. And I need to connect to users SFTP server and need to execute few commands like cd, rm, rename etc on remote server.

 

And to connect to that server I need to fulfill 2 step authentication (username/password and key authentication)

 

So if i ssh and provide key from shell script which will be mounted in docker. For that i will have to add the key needs to be installed in all nodes and then needs to be added to users sftp server.

 

It does not seem to be best solution

 

I there any other way where  I can use NiFi processor to execute these commands to remote sftp server which needs to be connected through 2 step authentication

 

Any advise will be very helpful

 

Thank you.

Shalaka

araujo
araujoAnswer
Platinum ⭐️⭐️⭐️
March 1, 2022

Hi @Elsaa ,

 

It's possible to do something similar by setting "Dot Rename = true" in the SFTP processor.

If that's set, while your file is being uploaded NiFi will create the file with a dot prefix(e.g.  .filename1.xml) and when the upload finishes it will rename the file to its original name.

 

araujo_0-1646119065498.png

 

Cheers,

André

 

 

-- Was your question answered? Please take some time to click on "Accept as Solution" below this post. If you find a reply useful, say thanks by clicking on the thumbs up button.
View in thread

7 replies

araujo
Platinum ⭐️⭐️⭐️
araujo
Platinum ⭐️⭐️⭐️
February 23, 2022

@Elsaa ,

 

If you can create a shell script that does what you need, you can use NiFi's ExecuteProcess to execute that shell script.

 

André

Translate
-- Was your question answered? Please take some time to click on "Accept as Solution" below this post. If you find a reply useful, say thanks by clicking on the thumbs up button.
    Elsaa
    ElsaaAuthor
    Silver ⭐️⭐️
    February 24, 2022

     

     

     

     

    @araujo 

    If I use shell script I will have to establish 2 step authentication connection from shell script to execute the commands to remote server.

    In that case I will have to add public key from all possible nodes (where my docker image will be installed) to remote sftp server. I believe it is not a good solution.

    To avoid this I wanted to use NiFi processor.

     

    2 possible scenarioes-

    1. If I can use NiFi processor to execute commands on remote sftp server directly

    2. If I can create keys and install it to node through docker image (so even if docker is intalled on another node the key will remain same which is added to remote server). In this case i can just use shell script which will establish connection and execute commands.

    Translate
    araujo
    Platinum ⭐️⭐️⭐️
    araujo
    Platinum ⭐️⭐️⭐️
    February 25, 2022

    I'm not sure if it's possible run SFTP commands other than put/get in NiFi.

     

    Sorry,

    André

     

    Translate
    -- Was your question answered? Please take some time to click on "Accept as Solution" below this post. If you find a reply useful, say thanks by clicking on the thumbs up button.
    Elsaa
    ElsaaAuthor
    Silver ⭐️⭐️
    March 3, 2022

    I want it to pass as filename.part  and rename to filename.xml

    I think I see one more option as temporary rename which renames file as mentioned name while transfer and then rename to original once file is transferred.

     

    Translate
    araujo
    Platinum ⭐️⭐️⭐️
    araujo
    Platinum ⭐️⭐️⭐️
    March 3, 2022

    That's true! That option fits perfectly to your needs.

     

    André

    Translate
    -- Was your question answered? Please take some time to click on "Accept as Solution" below this post. If you find a reply useful, say thanks by clicking on the thumbs up button.
      Terms & ConditionsAccessibility statement