Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

00zie -shell action--scp

Highlighted

00zie -shell action--scp

New Contributor

Hi Guys... Need your help regarding the oozie shell action to scp a file from remote server,here the main aim is to SCP log files from remote server to HDFS .

Lets say we have 3 servers - A,B,C-----A and B are part of the hadoop cluster ------c is remote server(outside cluster) Here, A is ssh passwordless to C(A's public key has been shared with C server)

Now from B datanode(some arbitatry node) we are trying to access C using Private key In above scenario B is random datanode picked by oozie server when running the shell action through oozie job.

NOTE: I DO NOT WANT TO USE SSH ACTION , THIS HAS TO OOZIE SHELL ACTION.

I know its little confusing but request you to please help. Also let me know if further details are required.

1. Shell script scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=known_hosts -i id_rsa user@wxXXXX:/home/USER/job.properties ${wd}/j1_ssh.properties

2. Workflow.xml <action name="shell_oozie"> <shell xmlns="uri:oozie:shell-action:0.2"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>oozie.launcher.mapred.job.queue.name</name> <value>${queueName}</value> </property> </configuration> <exec>scp_v3.sh</exec> <file>hdfs://XXXXXXXX/user/oozie_task/scp_v3.sh#scp_v3.sh</file> <file>hdfs://XXXXXXXX/user/oozie_task/id_rsa#id_rsa</file> <file>hdfs://XXXXXXXX/user/oozie_task/k1_hosts#k1_hosts</file> </shell> <ok to="end"/> <error to="kill"/> </action> 


3. Error Could not create directory '/home/XXXX/.ssh'. /grid/2/hadoop/yarn/local/usercache/user/appcache/application_1497692254655_6044/container_e281_1497692254655_6044_01_000002 /j1_ssh.properties : No such file or directory

Don't have an account?
Coming from Hortonworks? Activate your account here