Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Executing Shell/Python script to a remote machine in NiFi in HDF

avatar
Contributor

I have a 3 nodes HDF cluster(in AWS cloud) where NiFi is running across the cluster, i want to use a NiFi processor to trigger shell/python script on a remote machine (OnPremise) to perform certain action written in the shell/python script.

1 ACCEPTED SOLUTION

avatar
Master Mentor

@Praveen Singh

NiFi offers a variety of processors that support executing commands or scripts locally to the NiFi installation. In order to execute a script remotely to NiFi you would still need to have a local script that would be called by the processor that could then go perform that task.

Thank you,

Matt

View solution in original post

16 REPLIES 16

avatar
Master Mentor

@Praveen Singh

NiFi offers a variety of processors that support executing commands or scripts locally to the NiFi installation. In order to execute a script remotely to NiFi you would still need to have a local script that would be called by the processor that could then go perform that task.

Thank you,

Matt

avatar
Master Mentor

@Praveen Singh

Try using the "ExecuteProcess" processor.

PropertyValue
Commandssh
Command Arguments-i "<path to private key>" <user>@<remotehost> python <script> &

Thanks,

Matt

avatar
Contributor

Hey @Matt Clarke, Is there a way we can connect to the remote machine with password instead of key. i was able to run to do ssh (with password) from backend. Can i do it from "ExecuteProcess" processor. thanks.

avatar
Master Mentor

@Praveen Singh

You could install sshpass which would allow you to use a password in the ssh connection, but i strongly recommend against this approach. This requires you to set your password in plaintext in your NiFi processor which exposes it to anyone who has access to view that component.

Thanks,

Matt

avatar
Contributor

@Matt I have tried exploring Execute Script processor and Execute Process Processor but was not able to do ssh to remote machine. Could you provide more detail on this or help me with the processor and configurations.

Thanks

avatar
Master Guru

Can you ssh to the remote machine from the command line?

avatar

In addition to above answers, I've had success using the ExecuteStream processor to stream a script to a remote host and pull the results back into the Dataflow. So you could run it using parameters from the flow and without needing to deploying the script before calling it. In this example, we kinit and run a hive query against a remote kerberised cluster node.

Processor NameReplaceText
Search Value(?s:^.*$)
Replacement Valueklist -s || kinit user@DOMAIN.COM -k -t ~/user.keytab hive -e "Some Hive Query"
Replacement StrategyAlways Replace

Route into:

Processor NameExecuteStreamCommand
Command Arguments-o StrictHostKeyChecking=no user@hostname.fqdn
Command Pathssh

avatar
Contributor

Hey @Dan Chaffelson thanks for sharing, Can you help me with a Processor (with configuration) that can be used to trigger a python script on remote machine (by doing ssh) and pulling back the result to HDF cluster. I have used Execute process processor with below configs to trigger the python script on remote machine. ( As suggest by @Matt Clarke) and was able to run the python script.

PropertyValue
Commandssh
Command Arguments-i "<path to private key>" <user>@<remotehost> python <script> &

Now i want to pull back the output of python command (which is at /home/abc with 777 permission) to HDF cluster. Is there a processor to do that ?

avatar

If you use the setup above, put your shell script to be executed on the remote machine in the 'Replacement Value' of the ReplaceText processor. It will bring back everything printed to StdOut. What is your Python script doing? There may be a better solution we could suggest if we knew the outcome you were trying to achieve.