Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Resolve CredentialProvider API not working for Hadoop SSL Passwords

Highlighted

Resolve CredentialProvider API not working for Hadoop SSL Passwords

New Contributor

I have been trying to get the Hadoop CredentialProvider API to work in order to remove clear text passwords from the core-site.xml file due to security requirements.

System Info

HDP 2.6.3
Hadoop 2.7.3
RHEL 7

HDFS (namenode, datanodes) fail to start. I have been receiving the following error in the hdfs namenode logs:

java.io.IOException: keystore password was incorrect
        at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2015)
        at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:238)
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at org.mortbay.jetty.security.SslSelectChannelConnector.createSSLContext(SslSelectChannelConnector.java:641)
        at org.mortbay.jetty.security.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:613)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.Server.doStart(Server.java:235)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:938)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:170)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:933)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:746)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:992)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:976)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1701)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1769)
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded
        ... 17 more

My setup steps to use the CredentialProvider were as follows:

  1. Create credential store (jceks file) and associate alias/values
    $ hadoop credential create ssl.server.keystore.password -value actualpassword -provider localjceks://file/location/test.jceks
    
    $ hadoop credential create ssl.server.keystore.keypassword -value actualpassword -provider localjceks://file/location/test.jceks
  2. Modify core-site.xml to point at local jceks file. I use a localjceks instead of hdfs since the passwords are needed to setup hdfs encryption and must be available prior to hdfs being started
    hadoop.security.credential.provider.path = localjceks://file/location/test.jceks

  3. Copy the local jceks file to /location/test.jceks on all cluster instances
  4. Change password properties in core-site.xml to "none" so actual passwords are no longer stored in cleartext
    Changed ssl.server.keystore.password to "none"
    Changed ssl.server.keystore.keypassword to "none"

Since the default password of the jceks file is left at "none" as specified in hadoop source, at first I didn't set any information on the jceks file password. After not having success, I did create a text file with none as the content. I set hadoop.security.credstore.java-keystore-provider.password-file to the location of this text file. This did not change the error.

$ hadoop credential list #shows the correct credential provider and that the aliases do exist.


I've looked at the hadoop 2.7.3 source code, mainly the NameNode.java, NameNodeHttpServer.java, HttpServer2.java, and DFSConfigKeys.java. I followed the path where it does use the correct getPassword function that leverages the credentialprovider api. I've tried turning on higher levels of debugging, but haven't received any helpful output.

Any suggestion are greatly appreciated!