Posts: 66
Registered: ‎01-19-2017
Accepted Solution

OOzie shell action-scp with user

[ Edited ]

In oozie shell action I have written code to copy the files from remote server using scp command and I have enabled the password less access to remote server to the user who submit the workflow.


Below are the workflow.xml



<workflow-app name="oracle_log" xmlns="uri:oozie:workflow:0.4">
<start to="scp_copy"/>
<action name="scp_copy">
<shell xmlns="uri:oozie:shell-action:0.1">
<ok to="end"/>
<error to="kill"/>
<kill name="kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
<end name="end"/>


hadoop fs -mkdir /user/{user_name}/dd
scp -r user@xxx:/data/input/ddd.txt /home/{user_name}/

First line in the> It successfully creates the directory with the same user the worflow the submit.


But the second line always communicates to remote machine with  Yarn user..


Why It communicating as YARN user to the remote instead the user which i pass in 




Please kindly help me.





Posts: 1,673
Kudos: 329
Solutions: 263
Registered: ‎07-31-2013

Re: OOzie shell action-scp with user

In unsecured mode, all YARN container processes execute as the Linux local user "yarn". This cannot be changed unless you either enable Kerberos based security or explicitly turn on the LinuxContainerExecutor [1], which will also require ensuring that local Linux accounts exist for all job submitting user.

The HADOOP_USER_NAME value affects only 'hadoop' and other related Apache Hadoop/Ecosystem commands. Since the 'scp' program is not a Hadoop program, it does not get influenced by the username carried by that variable. It instead runs as the linux user that runs the shell script - which is "yarn" due to the above.

[1] - and 'Always Use Linux Container Executor' under CM -> YARN -> Configuration