Does anyone know if it is possible or when support will be added to transfer data from HDFS to S3 and vice-versa using AWS KMS for client side encryption? The transfer from S3 to HDFS could be utilizing distcp if I were looking for a bulk means to load data into the cluster as unencrypted. It looks to be supported by EMR. Any alternative approaches would be appreciated. https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-emrfs-encryption-cse.html https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html ... View more

I have been trying to get the Hadoop CredentialProvider API to work in order to remove clear text passwords from the core-site.xml file due to security requirements. System Info HDP 2.6.3 Hadoop 2.7.3 RHEL 7 HDFS (namenode, datanodes) fail to start. I have been receiving the following error in the hdfs namenode logs: java.io.IOException: keystore password was incorrect at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2015) at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:238) at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70) at java.security.KeyStore.load(KeyStore.java:1445) at org.mortbay.jetty.security.SslSelectChannelConnector.createSSLContext(SslSelectChannelConnector.java:641) at org.mortbay.jetty.security.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:613) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.Server.doStart(Server.java:235) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:938) at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:170) at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:933) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:746) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:992) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:976) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1701) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1769) Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded ... 17 more My setup steps to use the CredentialProvider were as follows: Create credential store (jceks file) and associate alias/values $ hadoop credential create ssl.server.keystore.password -value actualpassword -provider localjceks://file/location/test.jceks $ hadoop credential create ssl.server.keystore.keypassword -value actualpassword -provider localjceks://file/location/test.jceks Modify core-site.xml to point at local jceks file. I use a localjceks instead of hdfs since the passwords are needed to setup hdfs encryption and must be available prior to hdfs being started hadoop.security.credential.provider.path = localjceks://file/location/test.jceks Copy the local jceks file to /location/test.jceks on all cluster instances Change password properties in core-site.xml to "none" so actual passwords are no longer stored in cleartext Changed ssl.server.keystore.password to "none" Changed ssl.server.keystore.keypassword to "none" Since the default password of the jceks file is left at "none" as specified in hadoop source, at first I didn't set any information on the jceks file password. After not having success, I did create a text file with none as the content. I set hadoop.security.credstore.java-keystore-provider.password-file to the location of this text file. This did not change the error. $ hadoop credential list #shows the correct credential provider and that the aliases do exist. I've looked at the hadoop 2.7.3 source code, mainly the NameNode.java, NameNodeHttpServer.java, HttpServer2.java, and DFSConfigKeys.java. I followed the path where it does use the correct getPassword function that leverages the credentialprovider api. I've tried turning on higher levels of debugging, but haven't received any helpful output. Any suggestion are greatly appreciated! ... View more

Is there a way to verify RPC encryption in a kerberized cluster for HDFS and YARN ports? Running openssl gives me a "connection reset by peer message" and I suspect this might be because of kerberos authentication. ... View more

Is it possible to encrypt all communication from/to Zookeeper? I haven't seen any HDP documentation. ... View more

I've had trouble granting administrator permissions for Accumulo to a kerberos user on the HDP 2.6.1 VirtualBox sandbox. I stop all Accumulo services from Ambari and run the following command. $ sudo -u accumulo ACCUMULO_CONF_DIR=/etc/accumulo/conf/server accumulo init --reset-security -u me@EXAMPLE.COM I enter the passwords as blank since it's a kerberized cluster. The command outputs: 2017-11-07 09:46:26,012 [conf.AccumuloConfiguration] INFO : Loaded class : org.apache.accumulo.server.security.handler.KerberosAuthorizor 2017-11-07 09:46:26,023 [conf.AccumuloConfiguration] INFO : Loaded class : org.apache.accumulo.server.security.handler.KerberosAuthenticator 2017-11-07 09:46:26,031 [conf.AccumuloConfiguration] INFO : Loaded class : org.apache.accumulo.server.security.handler.KerberosPermissionHandler 2017-11-07 09:46:26,281 [handler.KerberosAuthenticator] INFO : Removed /accumulo/0c13d53b-b043-4887-a839-a6ee9749f919/users/ from zookeeper I start all Accumulo services from Ambari. I kinit with me@EXAMPLE.COM successfully. I run: $ accumulo shell me@EXAMPLE.COM@hdp-accumulo-instance> whoami #returns the correct user me@EXAMPLE.COM@hdp-accumulo-instance> userpermissions # shows that I only have read permissions on the following tables: Namespace permissions (accumulo): Namespace.READ Table permissions (accumulo.metadata): Table.READ Table permissions (accumulo.replication): Table.READ Table permissions (accumulo.root): Table.READ I am unable to perform any non-read actions on any tables with these permissions. me@EXAMPLE.COM@hdp-accumulo-instance> createtable test 2017-11-07 10:02:09,065 [shell.Shell] ERROR: org.apache.accumulo.core.client.AccumuloSecurityException: Error PERMISSION_DENIED for user me@EXAMPLE.COM on table test(?) - User does not have permission to perform this action me@EXAMPLE.COM@hdp-accumulo-instance> grant System.CREATE_TABLE -s -u me@EXAMPLE.COM 2017-11-07 10:05:17,661 [shell.Shell] ERROR: org.apache.accumulo.core.client.AccumuloSecurityException: Error PERMISSION_DENIED for user me@EXAMPLE.COM - User does not have permission to perform this action Appreciate any help! ... View more