Reply
Contributor
Posts: 29
Registered: ‎05-11-2016

Shell action in Oozie with copyFromLocal command

Hi All,

I am trying to run the shell from Oozie, the shell contain just simple copyFromLocal command.

The script is test.sh,

#!/bin/bash
hadoop fs -copyFromLocal /export/home/data/ /user/data/

(Local dir on Edge node= /export/home/data/

Hadoop dir = /user/data/)

 

This script is running fine on Edge node with expected result.

 

I have placed this script in workspace as well as mentioned in file option in workflow.

But I am getting below error,

  Log Type: stderr
  
  Log Length: 416
  copyFromLocal: `/export/home/data/': No such file or directory
  Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
  log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.impl.MetricsSystemImpl).
  log4j:WARN Please initialize the log4j system properly.
  log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

 Workflow.xml looks like below,

<workflow-app name="test_cpmv" xmlns="uri:oozie:workflow:0.4"> <start to="test_cpmv"/> <action name="test_cpmv" cred=""> <shell xmlns="uri:oozie:shell-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <exec>test.sh</exec> <file>test.sh#test.sh</file> </shell> <ok to="end"/> <error to="kill"/> </action> <kill name="kill"> <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="end"/> </workflow-app>
 
 
When I run the script on Edge node, it running fine may be because the path  /export/home/data/ is on Edge node. But when I try to run it from Oozie, I am not sure on which node it will run (error may be because of this reason).
 
So how can I fix this ?
Do I need to setup a reverse ssh i.e. from hadoop cluster to Edge node for Oozie ?
 
 
Thank you.
 
 
 
Posts: 1,836
Kudos: 416
Solutions: 295
Registered: ‎07-31-2013

Re: Shell action in Oozie with copyFromLocal command

> When I run the script on Edge node, it running fine may be because the path /export/home/data/ is on Edge node. But when I try to run it from Oozie, I am not sure on which node it will run (error may be because of this reason).

Yes, this is the cause. The shell action runs via an MR launch, so would run from a random NodeManager.

If you need to run something on a specific host, you should use the SSH action instead: http://archive.cloudera.com/cdh5/cdh/5/oozie/DG_SshActionExtension.html

The SSH action requires you to allow the "oozie" user to be able to SSH without a passphrase into the targeted user@host. The SSH action runs from within the Oozie server (not via an MR launch) so any SSH client-side configs you need to make, would go to your "oozie" user's home directory (typically /var/lib/oozie).
Highlighted
Contributor
Posts: 29
Registered: ‎05-11-2016

Re: Shell action in Oozie with copyFromLocal command

Thank you Harsh.

Though we don't have enterprise scheduler yet, I am trying to schedule most of jobs through Oozie.

But for data ingestion .sh scripts on Edge node, I may need to use a crontab now.

SSH setup may take long time.

 

Any suggestions are welcome other that cron and SSH.

Announcements