Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
Super Guru

This article is an update of one on http://hadooped.blogspot.com/2013/10/apache-oozie-part-13-oozie-ssh-action_30.html, authored by @Anagha Khanolkar.

Below are the steps to setup Oozie workflow using ssh-action:

Step 1. Create job.properties.

Example:

#************************************************* 
#  job.properties
#*************************************************
nameNode=hdfs://<namenode-machine-fqdn>:8020
jobTracker=<resource-manager-fqdn>:8050
queueName=default
oozie.libpath=${nameNode}/user/oozie/share/lib
oozie.use.system.libpath=true
oozie.wf.rerun.failnodes=true
oozieProjectRoot=${nameNode}/user/${user.name}
appPath=${oozieProjectRoot}
oozie.wf.application.path=${appPath}
inputDir=${oozieProjectRoot}
focusNodeLogin=<username>@<remote-host-where-you-have-your-shell-script(s)>
shellScriptPath=~/uploadFile.sh
emailToAddress=<email-id>

Step 2. Write workflow.xml

Example:

<!--******************************************-->
<!--workflow.xml                              -->
<!--******************************************-->
<workflow-app name="WorkFlowForSshAction" xmlns="uri:oozie:workflow:0.1">
    <start to="sshAction"/>
    <action name="sshAction">
        <ssh xmlns="uri:oozie:ssh-action:0.1">
            <host>${focusNodeLogin}</host>
       <command>${shellScriptPath}</command>
            <capture-output/>
        </ssh>
   <ok to="sendEmail"/>
        <error to="killAction"/>
    </action>
    <action name="sendEmail">
        <email xmlns="uri:oozie:email-action:0.1">
              <to>${emailToAddress}</to>
              <subject>Output of workflow ${wf:id()}</subject>
              <body>Status of the file move: ${wf:actionData('sshAction')['STATUS']}</body>
        </email>
        <ok to="end"/>
        <error to="end"/>
    </action>
    <kill name="killAction">
        <message>"Killed job due to error"</message>
    </kill>
    <end name="end"/>
</workflow-app>

Step 3. Write sample uploadFile.sh script

Example:

#!/bin/bash
hadoop fs -put ~/test /user/<username>/uploadedbyoozie

Step 4. Upload workflow.xml to ${appPath} defined in job.properties.

Step 5. Login to Oozie host by "oozie" user.

Step 6. Generate a key pair (if you don't have already ) using 'ssh-keygen' command

Step 7. On Oozie host copy ~/.ssh/id_rsa.pub and paste it on <remote-host>'s ~/.ssh/authorized_keys file (focus node)

Step 8. Test password-less ssh from oozie@oozie-host to <username>@<remote-host>

Step 9. if step 7 succeeds then go ahead and run oozie job. it should complete without error

Note - In order to get password-less ssh working please make sure that:

1. You have 700 permissions on ~/.ssh directory

2. 600 permissions on ~/.ssh/authorized_keys file on remote-host

3. 600 to ~/.ssh/id_rsa

4. 644 to ~/.ssh/id_rsa.pub

8,601 Views
Comments

Nice one!

Not applicable

Tried several things already but always ending up with the same error. The workflow errors out with: 2016-06-08 16:24:23,890 WARN ActionStartXCommand:523 - SERVER[hadwkr3-dg.ie.tslabs.hpecorp.net] USER[hdfs] GROUP[-] TOKEN[] APP[ssh-wf-test2] JOB[0000974-160503085957938-oozie-oozi-W] ACTION[0000974-160503085957938-oozie-oozi-W@ssh] Error starting action [ssh]. ErrorType [TRANSIENT], ErrorCode [FNF], Message [FNF: Required Local file /var/tmp/oozie/oozie-oozi1017005846702963868.dir/ssh/ssh-base.sh not present.] org.apache.oozie.action.ActionExecutorException: FNF: Required Local file /var/tmp/oozie/oozie-oozi1017005846702963868.dir/ssh/ssh-base.sh not present. at org.apache.oozie.action.ssh.SshActionExecutor.execute(SshActionExecutor.java:572) at org.apache.oozie.action.ssh.SshActionExecutor.start(SshActionExecutor.java:206) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64) at org.apache.oozie.command.XCommand.call(XCommand.java:286) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:321) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:250) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Required Local file /var/tmp/oozie/oozie-oozi1017005846702963868.dir/ssh/ssh-base.sh not present. at org.apache.oozie.action.ssh.SshActionExecutor.setupRemote(SshActionExecutor.java:367) at org.apache.oozie.action.ssh.SshActionExecutor$1.call(SshActionExecutor.java:208) at org.apache.oozie.action.ssh.SshActionExecutor$1.call(SshActionExecutor.java:206) at org.apache.oozie.action.ssh.SshActionExecutor.execute(SshActionExecutor.java:550) ... 10 more Have you encountered this or have a clue on how to workaround it? Using oozie 4.2.0 on hdp2.4

Not applicable

Also

  • Make sure that curl is installed in target machine
  • make sure that target machine can see oozie server and able to run the curl command against it

In my case I've got the timeout issue and installed curl but didn't work so I added the FQDN in /etc/hosts and it worked perfectly well.

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
1 of 1
Last update:
‎12-21-2015 06:03 AM
Updated by:
 
Contributors
Top Kudoed Authors