Executing Shell/Python script to a remote machine in NiFi in HDF

[#1]

I have a 3 nodes HDF cluster(in AWS cloud) where NiFi is running across the cluster, i want to use a NiFi processor to trigger shell/python script on a remote machine (OnPremise) to perform certain action written in the shell/python script.

[#2 · Accepted solution]

@Praveen Singh

NiFi offers a variety of processors that support executing commands or scripts locally to the NiFi installation. In order to execute a script remotely to NiFi you would still need to have a local script that would be called by the processor that could then go perform that task.

Thank you,

Matt

[#3]

@Praveen Singh

Try using the "ExecuteProcess" processor.

Property	Value
Command	ssh
Command Arguments	-i "<path to private key>" <user>@<remotehost> python <script> &

Thanks,

Matt

[#4]

Hey @Matt Clarke, Is there a way we can connect to the remote machine with password instead of key. i was able to run to do ssh (with password) from backend. Can i do it from "ExecuteProcess" processor. thanks.

[#5]

@Praveen Singh

You could install sshpass which would allow you to use a password in the ssh connection, but i strongly recommend against this approach. This requires you to set your password in plaintext in your NiFi processor which exposes it to anyone who has access to view that component.

Thanks,

Matt

[#6]

@Matt I have tried exploring Execute Script processor and Execute Process Processor but was not able to do ssh to remote machine. Could you provide more detail on this or help me with the processor and configurations.

Thanks

[#7]

Can you ssh to the remote machine from the command line?

[#8]

In addition to above answers, I've had success using the ExecuteStream processor to stream a script to a remote host and pull the results back into the Dataflow. So you could run it using parameters from the flow and without needing to deploying the script before calling it. In this example, we kinit and run a hive query against a remote kerberised cluster node.

Processor Name	ReplaceText
Search Value	(?s:^.*$)
Replacement Value	klist -s || kinit user@DOMAIN.COM -k -t ~/user.keytab hive -e "Some Hive Query"
Replacement Strategy	Always Replace

Route into:

Processor Name	ExecuteStreamCommand
Command Arguments	-o StrictHostKeyChecking=no user@hostname.fqdn
Command Path	ssh

[#9]

Hey @Dan Chaffelson thanks for sharing, Can you help me with a Processor (with configuration) that can be used to trigger a python script on remote machine (by doing ssh) and pulling back the result to HDF cluster. I have used Execute process processor with below configs to trigger the python script on remote machine. ( As suggest by @Matt Clarke) and was able to run the python script.

Property	Value
Command	ssh
Command Arguments	-i "<path to private key>" <user>@<remotehost> python <script> &

Now i want to pull back the output of python command (which is at /home/abc with 777 permission) to HDF cluster. Is there a processor to do that ?

[#10]

If you use the setup above, put your shell script to be executed on the remote machine in the 'Replacement Value' of the ReplaceText processor. It will bring back everything printed to StdOut. What is your Python script doing? There may be a better solution we could suggest if we knew the outcome you were trying to achieve.