Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (1)
avatar
Master Guru

This article is an update of one on http://hadooped.blogspot.com/2013/10/apache-oozie-part-13-oozie-ssh-action_30.html, authored by @Anagha Khanolkar.

Below are the steps to setup Oozie workflow using ssh-action:

Step 1. Create job.properties.

Example:

#************************************************* 
#  job.properties
#*************************************************
nameNode=hdfs://<namenode-machine-fqdn>:8020
jobTracker=<resource-manager-fqdn>:8050
queueName=default
oozie.libpath=${nameNode}/user/oozie/share/lib
oozie.use.system.libpath=true
oozie.wf.rerun.failnodes=true
oozieProjectRoot=${nameNode}/user/${user.name}
appPath=${oozieProjectRoot}
oozie.wf.application.path=${appPath}
inputDir=${oozieProjectRoot}
focusNodeLogin=<username>@<remote-host-where-you-have-your-shell-script(s)>
shellScriptPath=~/uploadFile.sh
emailToAddress=<email-id>

Step 2. Write workflow.xml

Example:

<!--******************************************-->
<!--workflow.xml                              -->
<!--******************************************-->
<workflow-app name="WorkFlowForSshAction" xmlns="uri:oozie:workflow:0.1">
    <start to="sshAction"/>
    <action name="sshAction">
        <ssh xmlns="uri:oozie:ssh-action:0.1">
            <host>${focusNodeLogin}</host>
       <command>${shellScriptPath}</command>
            <capture-output/>
        </ssh>
   <ok to="sendEmail"/>
        <error to="killAction"/>
    </action>
    <action name="sendEmail">
        <email xmlns="uri:oozie:email-action:0.1">
              <to>${emailToAddress}</to>
              <subject>Output of workflow ${wf:id()}</subject>
              <body>Status of the file move: ${wf:actionData('sshAction')['STATUS']}</body>
        </email>
        <ok to="end"/>
        <error to="end"/>
    </action>
    <kill name="killAction">
        <message>"Killed job due to error"</message>
    </kill>
    <end name="end"/>
</workflow-app>

Step 3. Write sample uploadFile.sh script

Example:

#!/bin/bash
hadoop fs -put ~/test /user/<username>/uploadedbyoozie

Step 4. Upload workflow.xml to ${appPath} defined in job.properties.

Step 5. Login to Oozie host by "oozie" user.

Step 6. Generate a key pair (if you don't have already ) using 'ssh-keygen' command

Step 7. On Oozie host copy ~/.ssh/id_rsa.pub and paste it on <remote-host>'s ~/.ssh/authorized_keys file (focus node)

Step 8. Test password-less ssh from oozie@oozie-host to <username>@<remote-host>

Step 9. if step 7 succeeds then go ahead and run oozie job. it should complete without error

Note - In order to get password-less ssh working please make sure that:

1. You have 700 permissions on ~/.ssh directory

2. 600 permissions on ~/.ssh/authorized_keys file on remote-host

3. 600 to ~/.ssh/id_rsa

4. 644 to ~/.ssh/id_rsa.pub

14,863 Views
Comments
avatar
Super Guru

Nice one!

avatar
New Contributor

Also

  • Make sure that curl is installed in target machine
  • make sure that target machine can see oozie server and able to run the curl command against it

In my case I've got the timeout issue and installed curl but didn't work so I added the FQDN in /etc/hosts and it worked perfectly well.

Version history
Last update:
‎12-21-2015 06:03 AM
Updated by:
Contributors