Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]
Created on 07-20-2021 02:13 AM - edited 09-16-2022 07:42 AM
I run hadoop with ldap
hadoop.security.group.mapping=org.apache.hadoop.security.LdapGroupsMapping
hadoop.security.group.mapping.ldap.bind.user=uid=ldapadmin,ou=people,dc=join,dc=com
hadoop.security.group.mapping.ldap.bind.password=00000
hadoop.security.group.mapping.ldap.base=dc=join,dc=com
hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=posixAccount)(uid={0}))
hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=posixGroup)
hadoop.security.group.mapping.ldap.search.attr.member=memberUid
hadoop.security.group.mapping.ldap.search.attr.group.name=cn
but I run spark,I get a WARN
(LdapGroupsMapping:290)2021-07-13 13:02:45,523 WARN - [pool-2-thread-4:] ~ Failed to get groups for user jztwk (retry=0) by javax.naming.OperationNotSupportedException: [LDAP: error code 53 - unauthenticated bind (DN with no password) disallowed]
and I ldapsearch -x -D "uid=ldapadmin,ou=people,dc=join,dc=com" -W -b "dc=join,dc=com"
can get result
hdfs groups yarn
yarn : hadoop spark yarn
so How can I fix it
CDH 6.3.2
Created 07-20-2021 07:57 PM
2021-07-21 10:16:13,691 WARN [main] org.apache.hadoop.security.LdapGroupsMapping: Exception while trying to get password for alias hadoop.security.group.mapping.ldap.bind.password: java.io.IOException: Configuration problem with provider path. at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2272) at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2191) at org.apache.hadoop.security.LdapGroupsMapping.getPassword(LdapGroupsMapping.java:719) at org.apache.hadoop.security.LdapGroupsMapping.setConf(LdapGroupsMapping.java:616) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) at org.apache.hadoop.security.Groups.<init>(Groups.java:106) at org.apache.hadoop.security.Groups.<init>(Groups.java:102) at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:451) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:352) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:314) at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1973) at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:743) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:693) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:604) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:461) Caused by: java.io.FileNotFoundException: /var/run/cloudera-scm-agent/process/5182-yarn-NODEMANAGER/creds.localjceks (Permission denied) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.getInputStreamForFile(LocalJavaKeyStoreProvider.java:83) at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.locateKeystore(AbstractJavaKeyStoreProvider.java:321) at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:86) at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.<init>(LocalJavaKeyStoreProvider.java:58) at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.<init>(LocalJavaKeyStoreProvider.java:50) at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider$Factory.createProvider(LocalJavaKeyStoreProvider.java:177) at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:73) at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2253)
Created 07-21-2021 12:11 AM
I give every node all creds.localjceks "chmod a+r"
I don't find
Caused by: java.io.FileNotFoundException: /var/run/cloudera-scm-agent/process/5182-yarn-NODEMANAGER/creds.localjceks (Permission denied)
but I aslo get
WARN security.LdapGroupsMapping: Failed to get groups for user jztwk (retry=1) by javax.naming.OperationNotSupportedException: [LDAP: error code 53 - unauthenticated bind (DN with no password) disallowed]
Created 07-21-2021 12:49 AM
in log I can see
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:23:44,433 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hive) returned [hive]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:24:02,669 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(impala) returned [hive, impala]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:24:06,305 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(oozie) returned [oozie]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:24:45,912 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(jztwk) returned [jztwk, admin]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:24:54,996 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(yarn) returned [hadoop, spark, yarn]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:25:04,573 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hue) returned [hue]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:27:29,074 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hdfs) returned [hadoop, hdfs]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:27:53,362 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(mapred) returned [hadoop, mapred]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:28:14,267 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hbase) returned [hbase]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:28:15,444 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(spark) returned [spark]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:29:43,423 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(oozie) returned [oozie]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:30:02,530 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(impala) returned [hive, impala]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:30:04,808 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hue) returned [hue]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:32:30,027 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hdfs) returned [hadoop, hdfs]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:33:03,374 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(mapred) returned [hadoop, mapred]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:33:14,274 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hbase) returned [hbase]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:33:16,064 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(spark) returned [spark]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:34:48,481 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(oozie) returned [oozie]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:35:04,814 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hue) returned [hue]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:36:02,528 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(impala) returned [hive, impala]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:36:46,230 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(jztwk) returned [jztwk, admin]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:37:30,043 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hdfs) returned [hadoop, hdfs]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:37:39,400 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hive) returned [hive]
./hadoop-cmf-hdfs-NAMENODE-bigdser5.log.out:2021-07-21 15:38:14,291 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(hbase) returned [hbase]
Created 07-21-2021 08:36 PM
my openldap can anonymous access
I remove
hadoop.security.group.mapping.ldap.bind.user
hadoop.security.group.mapping.ldap.bind.password
that I don't get WARN