- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Owner Group Write Permision to an HDFS path
- Labels:
-
HDFS
-
Hortonworks Data Platform (HDP)
Created on
‎02-10-2020
09:05 AM
- last edited on
‎02-10-2020
10:56 AM
by
ask_bill_brooks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have run the following test case several times and got the same result.
Context:
1. My HDP cluster uses the simple mode to determine user identity. Kerberos is not enabled.
2. Below is the permission on hdfs folder /data/test
drwxrwxr-x - hdfs hadoop 0 2020-02-07 13:33 /data/test
So hdfs (the super user) is the owner and hadoop is the owner group. Both the owner user and owner group has write permission on the /data/test folder.
Steps:
On an edge node, I used ID command to confirm that the logged on user "testuser" is in hadoop group.
$ id
uid=1018(testuser) gid=1003(hadoop) groups=1003(hadoop),10(wheel), 1002(hdfs) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
However, testuser still ran into "Permission Denied".
$ hadoop fs -put ./emptyfile1.txt /data/test
put: Permission denied: user=testuser, access=WRITE, inode="/data/test":hdfs:hadoop:drwxrwxr-x
Then I used hdfs account to changed the folder owner to be testuser.
$ hadoop fs -chown testuser /data/test
From the same edge node, now testuser ran put command successfully.
Here is my question: why testuser cannot write to the hdfs folder via the owner group permissions?
Created ‎02-19-2020 01:29 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Finally, I figured out what is going on.
The root cause is that, I only set up testuser on edge nodes, not the name node.
I looked into this page, https://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-common/GroupsMapping.html, which shows that "For HDFS, the mapping of users to groups is performed on the NameNode. Thus, the host system configuration of the NameNode determines the group mappings for the users."
After I created the user on the NameNode and ran the command
hdfs dfsadmin -refreshUserToGroupsMappings
the copy is successful and there is no permission-denied error.
Created ‎02-10-2020 11:49 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What's the result of below command with respect to test user?
id -Gn testuser
Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Created ‎02-10-2020 11:51 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@GangWar Here it is.
$ id -Gn testuser
hadoop wheel hdfs
Created ‎02-19-2020 01:29 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Finally, I figured out what is going on.
The root cause is that, I only set up testuser on edge nodes, not the name node.
I looked into this page, https://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-common/GroupsMapping.html, which shows that "For HDFS, the mapping of users to groups is performed on the NameNode. Thus, the host system configuration of the NameNode determines the group mappings for the users."
After I created the user on the NameNode and ran the command
hdfs dfsadmin -refreshUserToGroupsMappings
the copy is successful and there is no permission-denied error.
