I have been trying to get the Hadoop CredentialProvider API to work in order to remove clear text passwords from the core-site.xml file due to security requirements. System Info HDP 2.6.3 Hadoop 2.7.3 RHEL 7 HDFS (namenode, datanodes) fail to start. I have been receiving the following error in the hdfs namenode logs: java.io.IOException: keystore password was incorrect
at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2015)
at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:238)
at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
at java.security.KeyStore.load(KeyStore.java:1445)
at org.mortbay.jetty.security.SslSelectChannelConnector.createSSLContext(SslSelectChannelConnector.java:641)
at org.mortbay.jetty.security.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:613)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.Server.doStart(Server.java:235)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:938)
at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:170)
at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:933)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:746)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:992)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:976)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1701)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1769)
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded
... 17 more My setup steps to use the CredentialProvider were as follows: Create credential store (jceks file) and associate alias/values $ hadoop credential create ssl.server.keystore.password -value actualpassword -provider localjceks://file/location/test.jceks
$ hadoop credential create ssl.server.keystore.keypassword -value actualpassword -provider localjceks://file/location/test.jceks Modify core-site.xml to point at local jceks file. I use a localjceks instead of hdfs since the passwords are needed to setup hdfs encryption and must be available prior to hdfs being started hadoop.security.credential.provider.path = localjceks://file/location/test.jceks Copy the local jceks file to /location/test.jceks on all cluster instances Change password properties in core-site.xml to "none" so actual passwords are no longer stored in cleartext Changed ssl.server.keystore.password to "none" Changed ssl.server.keystore.keypassword to "none" Since the default password of the jceks file is left at "none" as specified in hadoop source, at first I didn't set any information on the jceks file password. After not having success, I did create a text file with none as the content. I set hadoop.security.credstore.java-keystore-provider.password-file to the location of this text file. This did not change the error. $ hadoop credential list #shows the correct credential provider and that the aliases do exist. I've looked at the hadoop 2.7.3 source code, mainly the NameNode.java, NameNodeHttpServer.java, HttpServer2.java, and DFSConfigKeys.java. I followed the path where it does use the correct getPassword function that leverages the credentialprovider api. I've tried turning on higher levels of debugging, but haven't received any helpful output. Any suggestion are greatly appreciated!
... View more
