Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

SFTP via Shell action in Oozie WF


SFTP via Shell action in Oozie WF


I tried running shell action in oozie workflow. In that shell script, I used sftp protocol using AWS pem key to connect to a EC2 Instance (to import files stored in this machine). I uploaded script and the pem key in HDFS (/user/admin/pig_scripts/) and also attached both explicitly in oozie workflow using files tags.


I tried running the above workflow but got the following host key authentication error. Being a newbie, I dont know much about this shell action in oozie.

Shell Script :


#Connect to FTP and get  files
ssh -i $KEY $USER@$HOST 'su -lc "cp -r '$RESULT' '$FILENAME_PATTERN3' ";' #-i $KEY $USER@$HOST
sftp -oIdentityFile=$KEY $USER@$HOST <<EOF

#Upload Files to HDFS filtered by name and size. Delete all files from Local FS afterwards.
for file in $FILENAME_PATTERN3
        FILESIZE=$(stat -c%s $file)
        if [ $FILESIZE -ge $SIZE_200MB ]; then
                hdfs dfs -put $file $HDFS_LOCATION



And I got the following error:



            Log Upload Time: Thu Mar 09 12:20:46 +0000 2017
            Log Length: 803
Host key verification failed.
Connecting to
Host key verification failed.
Couldn't read packet: Connection reset by peer
stat: cannot stat `us00*_2017-01-30.uf1': No such file or directory
./ line 33: [: -ge: unary operator expected
rm: cannot remove `us00*_2017-01-30.uf1': No such file or directory
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]



WF definition: 


<workflow-app name="Final_Workflow_ssh" xmlns="uri:oozie:workflow:0.5">
    <start to="shell-13ba"/>
    <kill name="Kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    <action name="shell-13ba">
        <shell xmlns="uri:oozie:shell-action:0.1">
        <ok to="End"/>
        <error to="Kill"/>
    <end name="End"/>


Kindly help me with the detailed steps to run shell action in oozie workflow.






Don't have an account?
Coming from Hortonworks? Activate your account here