question Files created after running oozie shell action are owned by yarn user in Support Questions

Files created after running oozie shell action are owned by yarn user

sshimpi — Fri, 03 Jun 2016 20:18:11 GMT

Hi,

I am running simple shell action using HUE(logged in as hdfs user in hue) -

$ cat test.sh
echo "hello" > /tmp/test

The workflow is getting executed successfully. When i check the files permission and ownership -

$ ls -al /tmp/test
-rw-r--r-- 1 yarn hadoop 6 2016-05-25 14:43 /tmp/test

The above output shows the file created via shell action has ownership as yarn.

How can I make oozie shell action to get the ownership to be same as the user who is running the "shell action/workflow"(in this case "hdfs")

So i am expecting output as shown below -

-rw-r--r-- 1 hdfs hadoop 6 2016-05-25 14:43 /tmp/test

Re: Files created after running oozie shell action are owned by yarn user

sunile_manjee — Fri, 03 Jun 2016 20:20:41 GMT

@Sagar Shimpi

You may need to enable proxyuser.

User ProxyUser Configuration

Oozie supports impersonation or proxyuser functionality (identical to Hadoop proxyuser capabilities and conceptually similar to Unix 'sudo').

Proxyuser enables other systems that are Oozie clients to submit jobs on behalf of other users.

Because proxyuser is a powerful capability, Oozie provides the following restriction capabilities (similar to Hadoop):

Proxyuser is an explicit configuration on per proxyuser user basis.
A proxyuser user can be restricted to impersonate other users from a set of hosts.
A proxyser user can be restricted to impersonate users belonging to a set of groups.

There are 2 configuration properties needed to set up a proxyuser:

oozie.service.ProxyUserService.proxyuser.#USER#.hosts: hosts from where the user #USER# can impersonate other users.
oozie.service.ProxyUserService.proxyuser.#USER#.groups: groups the users being impersonated by user #USER# must belong to.

Both properties support the '*' wildcard as value. Although this is recommended only for testing/development.

Re: Files created after running oozie shell action are owned by yarn user

sshimpi — Fri, 03 Jun 2016 21:04:52 GMT

@Sunile Manjee

I tried to set the property in oozie-site.xml with #user# as hdfs but still didnt worked.

Re: Files created after running oozie shell action are owned by yarn user

sunile_manjee — Fri, 03 Jun 2016 22:14:22 GMT

I assume restarted oozie?

Re: Files created after running oozie shell action are owned by yarn user

ibhatt — Fri, 03 Jun 2016 22:58:23 GMT

this is a known limitation in non-secure clusters, whereby the containers are running as YARN user and not running as logged user. try setting this

<env-var>HADOOP_USER_NAME=${wf:user()}</env-var>

Re: Files created after running oozie shell action are owned by yarn user

sshimpi — Mon, 06 Jun 2016 15:47:02 GMT

Yes. I did oozie restart after doing the modifications.

Re: Files created after running oozie shell action are owned by yarn user

sshimpi — Mon, 06 Jun 2016 15:47:59 GMT

@ibhatt

I already tried this but this didnt worked for me.

Re: Files created after running oozie shell action are owned by yarn user

quadoss — Tue, 07 Jun 2016 23:22:17 GMT

@Sagar Shimpi

By default the shell actions are not allowed to run as another user as sudo is blocked. If you want a yarn application to run as someone other than yarn (i.e. the submitter), then you need to enable the linux container executor so that the containers are started up by the submitting user. Also note the below setting information which also needs to be changed as well to achieve this.

With yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users=false (default), it runs as yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user (default is 'nobody')

With yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users=true, it runs as the user submitting the workflow.

Stating that there are issues around this also where it does not work as expected because of the issues https://issues.apache.org/jira/browse/YARN-2424

https://issues.apache.org/jira/browse/YARN-3462

The current suggestion that I can make is to add line to change the ownership of the file which was created using shell.

Re: Files created after running oozie shell action are owned by yarn user

sshimpi — Tue, 07 Jun 2016 23:37:09 GMT

Thanks for the info.