Exception while trying to get password for alias hadoop.security.group.mapping.ldap.bind.password:

[#1]

HI,  after i have integrated CDH with Openldap, I found there is  a WARNING in container log like below, try to get password file localjecks and permission denied.  

2022-03-31 00:53:13,420 WARN [main] org.apache.hadoop.security.LdapGroupsMapping: Exception while trying to get password for alias hadoop.security.group.mapping.ldap.ssl.keystore.password: 
java.io.IOException: Configuration problem with provider path.
	at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2118)
	at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2037)
	at org.apache.hadoop.security.LdapGroupsMapping.getPassword(LdapGroupsMapping.java:528)
	at org.apache.hadoop.security.LdapGroupsMapping.setConf(LdapGroupsMapping.java:473)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
	at org.apache.hadoop.security.Groups.<init>(Groups.java:104)
	at org.apache.hadoop.security.Groups.<init>(Groups.java:100)
	at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:435)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:341)
	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:308)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:895)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:861)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:728)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:387)
Caused by: java.io.FileNotFoundException: /run/cloudera-scm-agent/process/9392-yarn-NODEMANAGER/creds.localjceks (Permission denied)
	at java.io.FileInputStream.open0(Native Method)
	at java.io.FileInputStream.open(FileInputStream.java:195)
	at java.io.FileInputStream.<init>(FileInputStream.java:138)
	at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.getInputStreamForFile(LocalJavaKeyStoreProvider.java:83)
	at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.locateKeystore(AbstractJavaKeyStoreProvider.java:334)
	at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:88)
	at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.<init>(LocalJavaKeyStoreProvider.java:58)
	at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.<init>(LocalJavaKeyStoreProvider.java:50)
	at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider$Factory.createProvider(LocalJavaKeyStoreProvider.java:177)
	at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:73)
	at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2098)

this warning doesn't affect the mapreduce job, i just want to know how to resolve this.  

[#2]

@iamfromsky 

The path you are mentioning has permissions issues. As the root user can you 

# chmod 777 /run/cloudera-scm-agent/process/9392-yarn-NODEMANAGER/creds.localjceks

Then retry if that's successful then fine-tune the permissions .

Hope that helps

[#3]

as you know , this file locate many path, namenode, datenode, yarn ,hbase. and this file is created by CDH, do you suggest me to change these location path permission ? if i restart one of these role, this file as i think would created again , and the permission still would be 700

[#4]

@iamfromsky 

True let me check on that and revert on the config.

Can you share your integration steps or document?

[#5]

i followed https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_sg_ldap_grp_mappings.html#ldap_gr...    to set up openldap integration .

1.  install openldap

2.  set ldap parameter by doucments.

3. restart all service.

[#6]

@iamfromsky ,

Can you check if the yarn user belongs to the hadoop group in your machines (id yarn)? If not, try adding it to the group and check if it resolves your problem.

Cheers,

André

[#7]

[root@host243 ~]# id yarn
uid=979(yarn) gid=973(yarn) groups=973(yarn),982(hadoop),979(solr)
[root@host243 ~]# 
[root@host243 ~]# 
[root@host243 ~]# hdfs groups yarn
yarn : hadoop yarn

openldap user has been imported from OS user. so i think openldap user and group keep the same as os user/group.

there is just one think i'd like to share with you , after integrated with openldap, i haven't delete OS user. 

[#8]

@iamfromsky ,

What are the default permissions of the creds.localjceks file?

Cheers,

André

[#9]

/run/cloudera-scm-agent/process/9506-IMPALA-impala-CATALOGSERVER-45e2ae1dbc69e00f769182717dd71aa8-ImpalaRoleDiagnosticsCollection/creds.localjceks
/run/cloudera-scm-agent/process/9478-hue-KT_RENEWER/creds.localjceks
/run/cloudera-scm-agent/process/9476-hue-HUE_SERVER/creds.localjceks
/run/cloudera-scm-agent/process/9471-impala-CATALOGSERVER/creds.localjceks
/run/cloudera-scm-agent/process/9462-impala-CATALOGSERVER/creds.localjceks
/run/cloudera-scm-agent/process/9456-sentry-SENTRY_SERVER/creds.localjceks
/run/cloudera-scm-agent/process/9455-oozie-OOZIE_SERVER/creds.localjceks
/run/cloudera-scm-agent/process/9454-hue-KT_RENEWER/creds.localjceks
/run/cloudera-scm-agent/process/9452-hue-HUE_SERVER/creds.localjceks
/run/cloudera-scm-agent/process/9448-hive-HIVEMETASTORE/creds.localjceks
/run/cloudera-scm-agent/process/9446-sentry-SENTRY_SERVER/creds.localjceks
/run/cloudera-scm-agent/process/9445-oozie-OOZIE_SERVER/creds.localjceks
/run/cloudera-scm-agent/process/9444-hue-KT_RENEWER/creds.localjceks
/run/cloudera-scm-agent/process/9442-hue-HUE_SERVER/creds.localjceks
/run/cloudera-scm-agent/process/9438-hive-HIVEMETASTORE/creds.localjceks
/run/cloudera-scm-agent/process/9437-hue-KT_RENEWER/creds.localjceks
/run/cloudera-scm-agent/process/9435-hue-HUE_SERVER/creds.localjceks
/run/cloudera-scm-agent/process/9429-impala-CATALOGSERVER/creds.localjceks
/run/cloudera-scm-agent/process/9424-oozie-OOZIE_SERVER/creds.localjceks
/run/cloudera-scm-agent/process/9420-hive-HIVEMETASTORE/creds.localjceks
/run/cloudera-scm-agent/process/9400-sentry-SENTRY_SERVER/creds.localjceks
/run/cloudera-scm-agent/process/9399-yarn-RESOURCEMANAGER/creds.localjceks
/run/cloudera-scm-agent/process/9388-yarn-JOBHISTORY/creds.localjceks
/run/cloudera-scm-agent/process/9413-hbase-REGIONSERVER/creds.localjceks
/run/cloudera-scm-agent/process/9411-hbase-MASTER/creds.localjceks
/run/cloudera-scm-agent/process/9377-hdfs-NAMENODE-nnRpcWait/creds.localjceks
/run/cloudera-scm-agent/process/9361-hdfs-NAMENODE/creds.localjceks
/run/cloudera-scm-agent/process/9351-HBaseShutdown/creds.localjceks
/run/cloudera-scm-agent/process/9343-hue-HUE_SERVER/creds.localjceks
/run/cloudera-scm-agent/process/9345-hue-KT_RENEWER/creds.localjceks
/run/cloudera-scm-agent/process/9339-hive-HIVEMETASTORE/creds.localjceks
/run/cloudera-scm-agent/process/9338-oozie-OOZIE_SERVER/creds.localjceks
/run/cloudera-scm-agent/process/9337-sentry-SENTRY_SERVER/creds.localjceks
/run/cloudera-scm-agent/process/9333-hue-KT_RENEWER/creds.localjceks

every roles has their own creds.localjecks, and the default permission is 640. i pick some roles localjecks for your checking

[root@host21 ~]# ls -l /run/cloudera-scm-agent/process/9478-hue-KT_RENEWER/creds.localjceks
-rw-r----- 1 hue hue 1501 Mar 25 04:11 /run/cloudera-scm-agent/process/9478-hue-KT_RENEWER/creds.localjceks
[root@host21 ~]# ls -l /run/cloudera-scm-agent/process/9471-impala-CATALOGSERVER/creds.localjceks
-rw-r----- 1 impala impala 533 Mar 25 04:01 /run/cloudera-scm-agent/process/9471-impala-CATALOGSERVER/creds.localjceks
[root@host21 ~]# 
[root@host21 ~]# ls -l /run/cloudera-scm-agent/process/8788-hive-HIVEMETASTORE/creds.localjceks
-rw-r----- 1 hive hive 528 Mar  4 09:34 /run/cloudera-scm-agent/process/8788-hive-HIVEMETASTORE/creds.localjceks
[root@host21 ~]# ls -l /run/cloudera-scm-agent/process/9295-yarn-RESOURCEMANAGER/creds.localjceks
-rw-r----- 1 yarn hadoop 533 Mar 25 03:10 /run/cloudera-scm-agent/process/9295-yarn-RESOURCEMANAGER/creds.localjceks

when i run hive sql or sqoop , the permission denied of creds.localjecks happended.  

[#10]

@iamfromsky ,

Could you please share the output of this command:

ls -ln /run/cloudera-scm-agent/process/9295-yarn-RESOURCEMANAGER/creds.localjceks

Cheers,

André