How to run shell script on remote server

[#1]

Hi!!

I need to run a Linux command on list of files received from ListSFTP (most probably it should be run on remote FTP server) and pass forward the result of this operation as flow file to putHDFS.

Linux command is: 
cat ${LIST} | xargs -n 1 -I {} stdbuf -o 5GB awk -F'\x1b' '{OFS="|"};{print $1,$2,FILENAME}' {}

Can you please advise which processor can be most suitable.

Thanks in advance!

[#2 · Accepted solution]

@Gulshan Agivetova

The output of the ListSFTP processor will be a single 0 byte FlowFile for each file that is successfully listed based on the specific processor configuration. So I am not clear on where you get the value for ${list} from?

-

as far as your script is concerned, I would recommend putting those linux commands in a script on the target ftp server. You can then use ExecuteStreamCommand processor that can call the ssh command to remote in to your ftp server and execute the script containing the linux commands.

-

Thanks,

Matt

-

If you found this Answer addressed your original question, please take a moment to login and click "Accept" below the answer.

[#3]

@Matt Clarke

Thanks for your recommendation concerning running Linux commands!

Is there any processor that will output a list of all files in a specific directory? I would like to pass this list to Linux command.

[#4]

@Gulshan Agivetova

-

There is no NiFi processor that will produce a single NiFi FlowFile that contains a complete listing of all files in a specific target directory, but you can build a flow to do this.

The ListSFTP processor will produces one 0 byte FlowFile with the following attributes generated on each:

You could pass these 0 byte FlowFiles to a ReplaceText processor that could replace the 0 byte content with new content based off the values assigned to ${path}/${filename} for example. You could then feed all those FlowFiles to a MergeContent processor to merge them in to a single FlowFile with one path/filename per line. Then you can pass that merged file to the ExecuteStream Command processor.

-

Thanks,

Matt