WARN org.apache.hadoop.security.ShellBasedUnixGroupsMapping: unable to return groups for user host, PartialGroupNameException The user name 'host' is not found. id: host: no such user

[#1]

Have following error reported on Cloudera CM Node - 

/var/log/hadoop-hdfs/hadoop-cmf-hdfs-NAMENODE*.log.out

Need help in resolution.

2020-12-16 23:38:19,034 WARN org.apache.hadoop.security.ShellBasedUnixGroupsMapping: unable to return groups for user host
PartialGroupNameException The user name 'host' is not found. id: host: no such user
id: host: no such user

at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:212)
at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:133)
at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:72)
at org.apache.hadoop.security.Groups$GroupCacheLoader.fetchGroupList(Groups.java:368)
at org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:309)
at org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:267)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829)
at org.apache.hadoop.security.Groups.getGroups(Groups.java:225)
at org.apache.hadoop.security.UserGroupInformation.getGroups(UserGroupInformation.java:1778)
at org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1766)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.<init>(FSPermissionChecker.java:66)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.getPermissionChecker(FSDirectory.java:3468)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getPermissionChecker(FSNamesystem.java:4079)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:4269)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:901)

at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829)
at org.apache.hadoop.security.Groups.getGroups(Groups.java:225)
at org.apache.hadoop.security.UserGroupInformation.getGroups(UserGroupInformation.java:1778)
at org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1766)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.<init>(FSPermissionChecker.java:66)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.getPermissionChecker(FSDirectory.java:3468)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getPermissionChecker(FSNamesystem.java:4079)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:4269)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:901)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getFileInfo(AuthorizationProviderProxyClientProtocol.java:528)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:839)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)

[#2]

This could be as simple as a typo somewhere in your configs, but it's hard to tell where. Looks like the username was set to "host" somehow and hdfs can't authenticate that as a user. Without more context it's hard to say any more.

[#3]

Which config this could be ?
I can share more logs if you’d like to see. This started coming after I
added 6 nodes to scale.

[#4]

I tried to grep for 'user' or 'host in all configs under /etc/hadoop/conf.cloudera.hdfs and dont see any reference for that. Any direction here would be certainly helpful

Wondering if it could be due to following kerberos property - 

<property>
<name>dfs.namenode.kerberos.principal</name>
<value>hdfs/_HOST@MOB.NUANCE.COM</value>
</property>

[#5]

@aakulov - Did you get a chance to look at this one ? See anything concerning?

[#6]

Settings look fine. _HOST gets replaced by the actual FQDN of the host at runtime. One thing to check is to make sure reverse DNS lookup works on all hosts.