- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
Created on 12-21-2015 06:03 AM
This article is an update of one on http://hadooped.blogspot.com/2013/10/apache-oozie-part-13-oozie-ssh-action_30.html, authored by @Anagha Khanolkar.
Below are the steps to setup Oozie workflow using ssh-action:
Step 1. Create job.properties.
Example:
#************************************************* # job.properties #************************************************* nameNode=hdfs://<namenode-machine-fqdn>:8020 jobTracker=<resource-manager-fqdn>:8050 queueName=default oozie.libpath=${nameNode}/user/oozie/share/lib oozie.use.system.libpath=true oozie.wf.rerun.failnodes=true oozieProjectRoot=${nameNode}/user/${user.name} appPath=${oozieProjectRoot} oozie.wf.application.path=${appPath} inputDir=${oozieProjectRoot} focusNodeLogin=<username>@<remote-host-where-you-have-your-shell-script(s)> shellScriptPath=~/uploadFile.sh emailToAddress=<email-id>
Step 2. Write workflow.xml
Example:
<!--******************************************--> <!--workflow.xml --> <!--******************************************--> <workflow-app name="WorkFlowForSshAction" xmlns="uri:oozie:workflow:0.1"> <start to="sshAction"/> <action name="sshAction"> <ssh xmlns="uri:oozie:ssh-action:0.1"> <host>${focusNodeLogin}</host> <command>${shellScriptPath}</command> <capture-output/> </ssh> <ok to="sendEmail"/> <error to="killAction"/> </action> <action name="sendEmail"> <email xmlns="uri:oozie:email-action:0.1"> <to>${emailToAddress}</to> <subject>Output of workflow ${wf:id()}</subject> <body>Status of the file move: ${wf:actionData('sshAction')['STATUS']}</body> </email> <ok to="end"/> <error to="end"/> </action> <kill name="killAction"> <message>"Killed job due to error"</message> </kill> <end name="end"/> </workflow-app>
Step 3. Write sample uploadFile.sh script
Example:
#!/bin/bash hadoop fs -put ~/test /user/<username>/uploadedbyoozie
Step 4. Upload workflow.xml to ${appPath} defined in job.properties.
Step 5. Login to Oozie host by "oozie" user.
Step 6. Generate a key pair (if you don't have already ) using 'ssh-keygen' command
Step 7. On Oozie host copy ~/.ssh/id_rsa.pub and paste it on <remote-host>'s ~/.ssh/authorized_keys file (focus node)
Step 8. Test password-less ssh from oozie@oozie-host to <username>@<remote-host>
Step 9. if step 7 succeeds then go ahead and run oozie job. it should complete without error
Note - In order to get password-less ssh working please make sure that:
1. You have 700 permissions on ~/.ssh directory
2. 600 permissions on ~/.ssh/authorized_keys file on remote-host
3. 600 to ~/.ssh/id_rsa
4. 644 to ~/.ssh/id_rsa.pub
Created on 02-15-2016 03:17 PM
- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Report Inappropriate Content
Nice one!
Created on 09-06-2016 05:17 PM
- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Report Inappropriate Content
Also
- Make sure that curl is installed in target machine
- make sure that target machine can see oozie server and able to run the curl command against it
In my case I've got the timeout issue and installed curl but didn't work so I added the FQDN in /etc/hosts and it worked perfectly well.