Created 10-26-2016 09:56 AM
When I see the Name Node logs, I continuously see the message going on in the logs.
hadoop-hdfs-namenode-<HOST_NAME>.log
WARN security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1573)) - No groups available for user hdfs
I checked on the Name node the user hdfs does exist on the host and it is member of the gorup hadoop.
$id hdfs
uid=123(hdfs) gid=663(hadoop) groups=993(hadoop)
I am concerned because of three things.
1) Lack of useful information is hidden under spams
2) Possible performance hit due to I/O
3) Unnecessarily logs are getting bigger and occupying disk space.
What configuration do you think are we missing here?
Thanks,
SS
Created 10-26-2016 10:18 AM
@Smart Soultions
output of id hdfs should be
[root@hdp-2 ~]# id hdfs
uid=9022(hdfs) gid=501(hadoop) groups=501(hadoop),9014(hdfs)
In your output there is no group like hdfs
Create hdfs goup and add hdfs user in hdfs group. And try to restart namenode.
groupadd group1
usermod -a -G grpup1 abc
In this example I created group group1 adn addedd the user abc to group1.
Created 10-26-2016 10:18 AM
@Smart Soultions
output of id hdfs should be
[root@hdp-2 ~]# id hdfs
uid=9022(hdfs) gid=501(hadoop) groups=501(hadoop),9014(hdfs)
In your output there is no group like hdfs
Create hdfs goup and add hdfs user in hdfs group. And try to restart namenode.
groupadd group1
usermod -a -G grpup1 abc
In this example I created group group1 adn addedd the user abc to group1.
Created 10-26-2016 10:52 AM
Thank you @Ashnee Sharma Is it a work around or hdfs user needs to be in the hdfs group?
I also checked in the log, for the spark user it says no groups found for user spark, for user yarn same is the case.
Do you think we need to create groups for each user?
Thank you.
Created 10-26-2016 11:33 AM
yes you need to create group of each user. If groups are not there.
Please accept the ans so it will helps to others.
Created 10-26-2016 12:50 PM
Before accepting I am trying to understand the root cause, why I do not have groups for the users on my cluster.
Created 10-26-2016 03:16 PM
@Smart Solutions
May be some one deleted the group or not installed the properly. Because while installing HDP . Ambari creates user and group itself.
Created 10-27-2016 08:23 AM
@Ashnee Sharma No one deleted the groups. Well digging further found this issue is different from deletion of the groups. Its something related with infra initial setup 🙂 Anyway accepting the answer as it can help with normal scenarios.
Thank you.
Created 11-01-2016 10:28 PM
This error has nothing to do with the missing user-specific group. The NameNode has no requirement that the group name match the user name. As long as the id command succeeded and returned any groups the NameNode would not have logged the error.
This error was likely due to a temporary infrastructure issue as pointed out by @Smart Solutions in a later comment. The following article is a good starting point for any issues related to groups lookups.
https://community.hortonworks.com/articles/38591/hadoop-and-ldap-usage-load-patterns-and-tuning.html
Created 11-03-2016 12:02 PM
You are right Arpit. It is a deeper issue, related to groups in LDAP.