HDFS usermod -G

[#1]

Hello,

I  want to do a folder that can be accessed in r, w,x by all the users of the group 'developer' that I created.

I also want that all the folders/files created by these users in this folder can be accessed in r,w,x by the users from this group. 

All the other users should access the folder in r and x.

What should I do?

I tried:

- to create my folder with these rights: drwxrwxr-t+  

- the folder that I created is owned by admin who is part of the developer group.

All I think is missing is to add the default group of my users the developer group : do something like

hadoop def -usermod -G developer Peter

But this is not working!

Thank you!

[#2]

To begin with, my explanation will cover the case that you are using simple
authentication and the default Hadoop group mappings.

Hadoop does not use uid or gid.

You username on the system is passed to the namenode when you make a client
call.

hadoop fs -ls /

If your username is 'bob', the NameNode will receive the string "bob" to
identify you.
If your username is 'hdfs', the NameNode will receive the string "hdfs" to
identify you.

User and Group lookups are done by the NameNode.

The default for the property hadoop.security.group.mapping
is org.apache.hadoop.security.ShellBasedUnixGroupsMapping, which means a
username passed with an HDFS client API will do the equivalent of "id -Gn
" on the NameNode host.

On the host running the NameNode, use the standard unix commands to assign
users to groups.

$ sudo usermod -G developer Peter

You only need to define the group membership on the NameNode's host
operating system. Now if a user named 'Peter' exists on any other system
configured as an HDFS client, they will still be considered part of the
"developer" group by HDFS.

"hdfs groups " is a handy command you can run to see if a user
belongs to a group in HDFS

Create your directory in HDFS:

hadoop fs -mkdir /projects/only/developers
hadoop fs -chown anyuser:developers /projects/only/developers
hadoop fs -chmod 775 /projects/only/developers

Your directory is now writable by anyone in the group "developers" as
defined on the NameNode.

[#3]

Thank you david for this complete answer!

However, I'm facing some problems...

On my namenode, when I do 

sudo usermod -G developer Peter

I get user 'Peter' does not exist. (Note, I created this user with the Hue interface)

When I do 

hdfs groups Peter

 I get no group. To be precise I get

Peter:

I checked that the property hadoop.security.group.mapping is org.apache.hadoop.security.ShellBasedUnixGroupsMapping

Note: I created the user, the group and associated the group to the user in Hue interface. And these users, and groups do not seem to be visible on the OS (CentOS).

[#4]

Note:

- I tried the command

sudo usermod -G developer Peter

on all instances of the cluster => no result

- I tried to do the same thing on a quickstart VM and I managed to make it work by only using the interface. I still not manage to make it work on the 7 machine cluster.

- When we installed the cluster, we had to create a user in cloudera manager and then create them again in cloudera director. is this normal? can this be linked to my problem?

[#5 · Accepted solution]

Creating a user in Hue creates the user only in the Hue User database.

Creating a user in Cloudera Manager create the user only the the Cloudera
Manager User table.

Both the user and the groups need to exist on the NameNode host operating
system:

sudo useradd Peter
sudo usermod -G developer Peter

If you don't want the user to be able to log in to the NameNode:

sudo usermod -s /bin/falso Peter
or
sudo usermod -s /usr/bin/nologin Peter

[#6]

Adding onto David's reply, your end-goal for checking if HDFS understands your group setups, should be to have the command "hdfs groups <username>" return a satisfactory result.

Also checkout this blog post: http://blog.cloudera.com/blog/2012/03/authorization-and-authentication-in-hadoop/, and the general HDFS permissions guide: http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.h... for more reading on the topic.