Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to create group & assign user in HDFS

avatar
New Contributor

I have user LDAP and I want to add my user to new group in HDFS. By default, the group belongs to LDAP user has the same name with the user name itself. Example : my LDAP user is monica.baniwijaya and the user belongs to group monica.baniwijaya. I need to create new group and add my user to the group. I've tried this code below :

$ groupadd ARMIS
$ useradd –g ARMIS monica.baniwijaya

 and then :

$ su hdfs
$ hdfs dfsadmin -refreshUserToGroupsMappings

 

In HDFS, my user don't belongs to group ARMIS, but in CentOS, my user belongs to that group.

I have read many documentations about creating group in HDFS, and many articles said when I do code above, group will be created on HDFS cluster too.

Please help me to resolve this issue.

 

Note :

This is the current HDFS Configuration for hadoop.security.group.mapping : 

Capture.PNG

 

 

 

2 REPLIES 2

avatar
Explorer

Check whether you have write permissions to add an user and also share your core-site.xml conf.

avatar
Master Guru

@monicbw,

 

Hadoop uses the following to derive group membership when using ShellBasedUnixGroupsMapping:

 

id -gn monica.baniwijaya;id -Gn monica.baniwijaya

You can run the above to see what hdfs sees when it tries to figure out to what groups you belong.

Also, you can run the following to find out what groups hdfs thinks you belong to:

hdfs groups monica.baniwijaya

This will return what is in hdfs's user/group mapping cache.

Knowing what each of the above returns will go a long way toward understanding the problem better.

 

That said, I am concerned about 2 things you mentioned:
 (1)

 

You referred to "LDAP".  OS users and LDAP users are completely different unless you have some way for your OS to pull information from LDAP and return it for user/group requests (SSSD, IPA, Centrify, are a few examples).  How is LDAP involved in this situation?

 

(2)

in your "useradd" example, you are setting the primary group for your user which is not common as far as I know.  I think you meant to use -G instead which will add a secondary group to a user:

 -g, --gid GROUP name or ID of the primary group of the new
account
-G, --groups GROUPS list of supplementary groups of the new
account

Lastly, you mentioned that in HDFS your user belongs to a different group than on the OS and that you would expect the group to be created in HDFS too.  HDFS relies on the user/groups mapping returned from the OS... it will cache that information, but it is not created in HDFS.  Minor difference.  What test did you do to determine that the OS and HDFS groups appear different for your user?  I'd use the "id" and "hdfs groups" commands I pointed out earlier to get a better sense of what HDFS sees.