Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

SFTP via Shell action in Oozie WF

Highlighted

SFTP via Shell action in Oozie WF

Explorer

I tried running shell action in oozie workflow. In that shell script, I used sftp protocol using AWS pem key to connect to a EC2 Instance (to import files stored in this machine). I uploaded script and the pem key in HDFS (/user/admin/pig_scripts/) and also attached both explicitly in oozie workflow using files tags.

 

I tried running the above workflow but got the following host key authentication error. Being a newbie, I dont know much about this shell action in oozie.

Shell Script :

 

#Connect to FTP and get  files
cd $LOCALFS_TMP
ssh -i $KEY $USER@$HOST 'su -lc "cp -r '$RESULT' '$FILENAME_PATTERN3' ";' #-i $KEY $USER@$HOST
sftp -oIdentityFile=$KEY $USER@$HOST <<EOF
get /tmp/$FILENAME_PATTERN3
exit
EOF


#Upload Files to HDFS filtered by name and size. Delete all files from Local FS afterwards.
for file in $FILENAME_PATTERN3
do
        FILESIZE=$(stat -c%s $file)
        if [ $FILESIZE -ge $SIZE_200MB ]; then
                hdfs dfs -put $file $HDFS_LOCATION
        fi
done
rm $FILENAME_PATTERN3

 

 

And I got the following error:

 

 

          
            Log Upload Time: Thu Mar 09 12:20:46 +0000 2017
          
            Log Length: 803
          
Host key verification failed.
Connecting to ec2-xxxx.us-west-2.compute.amazonaws.com...
Host key verification failed.
Couldn't read packet: Connection reset by peer
stat: cannot stat `us00*_2017-01-30.uf1': No such file or directory
./try_Consolidated_v1.sh: line 33: [: -ge: unary operator expected
rm: cannot remove `us00*_2017-01-30.uf1': No such file or directory
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

 

 

WF definition: 

 

<workflow-app name="Final_Workflow_ssh" xmlns="uri:oozie:workflow:0.5">
    <start to="shell-13ba"/>
    <kill name="Kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <action name="shell-13ba">
        <shell xmlns="uri:oozie:shell-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <exec>/user/admin/pig_scripts/try_Consolidated_v1.sh</exec>
            <file>/user/admin/pig_scripts/try_Consolidated_v1.sh#try_Consolidated_v1.sh</file>
            <file>/user/admin/pig_scripts/hadoop-key.pem#hadoop-key.pem</file>
        </shell>
        <ok to="End"/>
        <error to="Kill"/>
    </action>
    <end name="End"/>
</workflow-app>

 

Kindly help me with the detailed steps to run shell action in oozie workflow.

 

 

Regards,

Anirban. 

 

Don't have an account?
Coming from Hortonworks? Activate your account here