question HDFS file/dir ownership to be <user>:<corresponding_group> in stead of <user>:supergroup in Support Questions

HDFS file/dir ownership to be : in stead of :supergroup

banshidhar_saho — Wed, 04 Nov 2020 13:08:57 GMT

Hi All,

In our CDP 7.1.2 cluster, default ownership of any file or directory is <user>and supergroup.

One application team wants their data owner group to be their own group in stead of supergroup.

for example:

Let the app team user is 'abc' and group is 'abc'.

Currently owner and group for any file/dir in HDFS is abc:supergroup.

They want it to be abc:abc.

Please suggest how to achieve this so that any new file created by abc user will have ownership as abc:abc.

Regards,

Banshi.

Re: HDFS file/dir ownership to be : in stead of :supergroup

tusharkathpal — Wed, 04 Nov 2020 14:51:12 GMT

@banshidhar_saho

This is the expected behavior as in Hadoop there are no groups based on user name. It does not use the OS level groups, instead it inherits it from parent directory. The reason is that group resolution happens on NameNode where groups related to the user may not exist.

To fix the case in your environment, modify the owner of the parent directory. This will cause new files to use the correct group.
Run:-->hdfs dfs -chown -R username:groupname <PATH to DIR>
This will recursively change the owner and group for the given <PATH>. (Note: do this with hdfs user)

You can further use acls' to ensure the groups you are trying have access to it.

hdfs dfs -setfacl -m default:group:<group-name>:rwx <Path>

Let me know if this helps

Re: HDFS file/dir ownership to be : in stead of :supergroup

banshidhar_saho — Wed, 04 Nov 2020 16:30:06 GMT

Hi @tusharkathpal ,

Thank you very much. I tested it. It worked for ownership.

I want 750 permission on the directory recursively.

However it's not reflecting in case of new file.

Is there a way to do that? Idea is to restrict 'others' from accessing the file/dir.

Regards,

Banshi.

Re: HDFS file/dir ownership to be : in stead of :supergroup

tusharkathpal — Wed, 04 Nov 2020 16:51:18 GMT

Hey @banshidhar_saho

Glad to hear it works.

regarding permission, it depends upon acl permissions you give. In the previous comment I had updated rwx for group. You can set it something for "other"

hdfs dfs -setfacl -m default:group:<group-name>:rwx <Path>

hdfs dfs -setfacl -m default:other::- <path>

Can you try the above and update if it works.

Re: HDFS file/dir ownership to be : in stead of :supergroup

Shelton — Wed, 04 Nov 2020 19:59:24 GMT

@banshidhar_saho

If you want to restrict the [others] you will need to apply POSIX permission on the file/directory look at the below examples

Below gives read and execute to [others]

$ hdfs dfs -setfacl -m others::r-x /path/to/dir

Below gives read to [others]

$ hdfs dfs -setfacl -m others::r-- /path/to/dir

Below gives no permissions to [others]

$ hdfs dfs -setfacl -m others::--- /path/to/dir

After the changes to validate run this snippet that should show it's set and to prove as [others]

$ hdfs dfs -getfacl /path/to/dir

try to list files in the directory

Happy hadooping

Re: HDFS file/dir ownership to be : in stead of :supergroup

banshidhar_saho — Thu, 05 Nov 2020 07:02:58 GMT

Hi @Shelton,

Thank you very much for sharing the exact commands.

Only thing that needs to change in the shared commands is - others should be other

Here is the command that worked for me.

$ hdfs dfs -setfacl -m other::--- /path/to/dir

Regards,

Banshi.

Re: HDFS file/dir ownership to be : in stead of :supergroup

Shelton — Thu, 05 Nov 2020 07:16:03 GMT

@banshidhar_saho

Surely that was I typo error I am happy that it worked for you. If you feel that helped you solve the issue you have, it would be great to accept it as an answer so the thread can be marked a solution for other members to reference.

Happy hadooping

Re: HDFS file/dir ownership to be : in stead of :supergroup

VidyaSargur — Thu, 05 Nov 2020 10:04:44 GMT

Hi @banshidhar_saho , I'm happy to see you resolved your issue. Can you please mark the appropriate reply as the solution? It will surely make it easier for others to find the answer in the future.

Re: HDFS file/dir ownership to be : in stead of :supergroup

banshidhar_saho — Thu, 05 Nov 2020 18:32:48 GMT

Hi Team,

Thank you very much for promptly responding and providing useful solution/direction.

While testing, app team reported that child dir/files are having r-x (for dir) and r--(files) for Others.

Looks like ACL are not getting inherited to child dir and files.

Can you confirm if ACL should inherit to child dir and files or not?

Is there any other configuration that I need to check and modify?

Regards,

Banshi.