Created on 07-20-2021 02:13 AM - edited 09-16-2022 07:42 AM
I run hadoop with ldap
hadoop.security.group.mapping=org.apache.hadoop.security.LdapGroupsMapping
hadoop.security.group.mapping.ldap.bind.user=uid=ldapadmin,ou=people,dc=join,dc=com
hadoop.security.group.mapping.ldap.bind.password=00000
hadoop.security.group.mapping.ldap.base=dc=join,dc=com
hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=posixAccount)(uid={0}))
hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=posixGroup)
hadoop.security.group.mapping.ldap.search.attr.member=memberUid
hadoop.security.group.mapping.ldap.search.attr.group.name=cn
but I run spark,I get a WARN
(LdapGroupsMapping:290)2021-07-13 13:02:45,523 WARN - [pool-2-thread-4:] ~ Failed to get groups for user jztwk (retry=0) by javax.naming.OperationNotSupportedException: [LDAP: error code 53 - unauthenticated bind (DN with no password) disallowed]
and I ldapsearch -x -D "uid=ldapadmin,ou=people,dc=join,dc=com" -W -b "dc=join,dc=com"
can get result
hdfs groups yarn
yarn : hadoop spark yarn
so How can I fix it
CDH 6.3.2
Created 07-20-2021 07:57 PM
2021-07-21 10:16:13,691 WARN [main] org.apache.hadoop.security.LdapGroupsMapping: Exception while trying to get password for alias hadoop.security.group.mapping.ldap.bind.password: java.io.IOException: Configuration problem with provider path. at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2272) at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2191) at org.apache.hadoop.security.LdapGroupsMapping.getPassword(LdapGroupsMapping.java:719) at org.apache.hadoop.security.LdapGroupsMapping.setConf(LdapGroupsMapping.java:616) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) at org.apache.hadoop.security.Groups.<init>(Groups.java:106) at org.apache.hadoop.security.Groups.<init>(Groups.java:102) at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:451) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:352) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:314) at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1973) at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:743) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:693) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:604) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:461) Caused by: java.io.FileNotFoundException: /var/run/cloudera-scm-agent/process/5182-yarn-NODEMANAGER/creds.localjceks (Permission denied) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.getInputStreamForFile(LocalJavaKeyStoreProvider.java:83) at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.locateKeystore(AbstractJavaKeyStoreProvider.java:321) at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:86) at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.<init>(LocalJavaKeyStoreProvider.java:58) at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.<init>(LocalJavaKeyStoreProvider.java:50) at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider$Factory.createProvider(LocalJavaKeyStoreProvider.java:177) at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:73) at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2253)
Created 07-21-2021 12:11 AM
I give every node all creds.localjceks "chmod a+r"
I don't find
Caused by: java.io.FileNotFoundException: /var/run/cloudera-scm-agent/process/5182-yarn-NODEMANAGER/creds.localjceks (Permission denied)
but I aslo get
WARN security.LdapGroupsMapping: Failed to get groups for user jztwk (retry=1) by javax.naming.OperationNotSupportedException: [LDAP: error code 53 - unauthenticated bind (DN with no password) disallowed]
Created 07-21-2021 12:49 AM
in log I can see
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:23:44,433 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hive) returned [hive]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:24:02,669 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(impala) returned [hive, impala]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:24:06,305 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(oozie) returned [oozie]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:24:45,912 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(jztwk) returned [jztwk, admin]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:24:54,996 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(yarn) returned [hadoop, spark, yarn]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:25:04,573 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hue) returned [hue]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:27:29,074 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hdfs) returned [hadoop, hdfs]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:27:53,362 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(mapred) returned [hadoop, mapred]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:28:14,267 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hbase) returned [hbase]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:28:15,444 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(spark) returned [spark]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:29:43,423 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(oozie) returned [oozie]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:30:02,530 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(impala) returned [hive, impala]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:30:04,808 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hue) returned [hue]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:32:30,027 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hdfs) returned [hadoop, hdfs]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:33:03,374 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(mapred) returned [hadoop, mapred]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:33:14,274 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hbase) returned [hbase]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:33:16,064 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(spark) returned [spark]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:34:48,481 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(oozie) returned [oozie]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:35:04,814 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hue) returned [hue]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:36:02,528 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(impala) returned [hive, impala]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:36:46,230 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(jztwk) returned [jztwk, admin]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:37:30,043 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hdfs) returned [hadoop, hdfs]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:37:39,400 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hive) returned [hive]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:38:14,291 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hbase) returned [hbase]
Created 07-21-2021 08:36 PM
my openldap can anonymous access
I remove
hadoop.security.group.mapping.ldap.bind.user
hadoop.security.group.mapping.ldap.bind.password
that I don't get WARN