- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Files created after running oozie shell action are owned by yarn user
- Labels:
-
Apache Hadoop
-
Apache Oozie
-
Cloudera Hue
Created 06-03-2016 01:18 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
I am running simple shell action using HUE(logged in as hdfs user in hue) -
$ cat test.sh echo "hello" > /tmp/test
The workflow is getting executed successfully. When i check the files permission and ownership -
$ ls -al /tmp/test -rw-r--r-- 1 yarn hadoop 6 2016-05-25 14:43 /tmp/test
The above output shows the file created via shell action has ownership as yarn.
How can I make oozie shell action to get the ownership to be same as the user who is running the "shell action/workflow"(in this case "hdfs")
So i am expecting output as shown below -
-rw-r--r-- 1 hdfs hadoop 6 2016-05-25 14:43 /tmp/test
Created 06-07-2016 04:22 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
By default the shell actions are not allowed to run as another user as sudo is blocked. If you want a yarn application to run as someone other than yarn (i.e. the submitter), then you need to enable the linux container executor so that the containers are started up by the submitting user. Also note the below setting information which also needs to be changed as well to achieve this.
With yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users=false (default), it runs as yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user (default is 'nobody')
With yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users=true, it runs as the user submitting the workflow.
Stating that there are issues around this also where it does not work as expected because of the issues https://issues.apache.org/jira/browse/YARN-2424
https://issues.apache.org/jira/browse/YARN-3462
The current suggestion that I can make is to add line to change the ownership of the file which was created using shell.
Created 06-03-2016 01:20 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Sagar Shimpi
You may need to enable proxyuser.
User ProxyUser Configuration
Oozie supports impersonation or proxyuser functionality (identical to Hadoop proxyuser capabilities and conceptually similar to Unix 'sudo').
Proxyuser enables other systems that are Oozie clients to submit jobs on behalf of other users.
Because proxyuser is a powerful capability, Oozie provides the following restriction capabilities (similar to Hadoop):
- Proxyuser is an explicit configuration on per proxyuser user basis.
- A proxyuser user can be restricted to impersonate other users from a set of hosts.
- A proxyser user can be restricted to impersonate users belonging to a set of groups.
There are 2 configuration properties needed to set up a proxyuser:
- oozie.service.ProxyUserService.proxyuser.#USER#.hosts: hosts from where the user #USER# can impersonate other users.
- oozie.service.ProxyUserService.proxyuser.#USER#.groups: groups the users being impersonated by user #USER# must belong to.
Both properties support the '*' wildcard as value. Although this is recommended only for testing/development.
Created 06-03-2016 02:04 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I tried to set the property in oozie-site.xml with #user# as hdfs but still didnt worked.
Created 06-03-2016 03:14 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I assume restarted oozie?
Created 06-06-2016 08:47 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes. I did oozie restart after doing the modifications.
Created 06-03-2016 03:58 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
this is a known limitation in non-secure clusters, whereby the containers are running as YARN user and not running as logged user. try setting this
<env-var>HADOOP_USER_NAME=${wf:user()}</env-var>
Created 06-06-2016 08:47 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I already tried this but this didnt worked for me.
Created 06-07-2016 04:22 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
By default the shell actions are not allowed to run as another user as sudo is blocked. If you want a yarn application to run as someone other than yarn (i.e. the submitter), then you need to enable the linux container executor so that the containers are started up by the submitting user. Also note the below setting information which also needs to be changed as well to achieve this.
With yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users=false (default), it runs as yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user (default is 'nobody')
With yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users=true, it runs as the user submitting the workflow.
Stating that there are issues around this also where it does not work as expected because of the issues https://issues.apache.org/jira/browse/YARN-2424
https://issues.apache.org/jira/browse/YARN-3462
The current suggestion that I can make is to add line to change the ownership of the file which was created using shell.
Created 06-07-2016 04:37 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for the info.