Does Vormetric work with Hortonworks over S3? We cannot use Amazon's KMS. Is there another solution that maintains keys entirely under the user's control?
Apache Ranger provides KMS services for Hortonworks Data Platform:
Vormetric is also a Hortonworks partner:
Yes, thanks--aware of Ranger, which would be ideal for our purposes, but it's not easy to tell from the literature whether Ranger can manage encryption at rest and key management on S3. The literature talks about HDFS, but I don't see any reference to S3 and have been told that it does not. Do all the Ranger features work for S3?
Some relevant details: This is an analytics application with all the data uploaded into S3. Hive runs directly against the S3 data, i.e, we do not move the data into HDFS for processing, If we worked on HDFS we could just use Ranger, but we need both the capacity and the durability provided by S3. Amazon's KMS is logically adequate to our needs, but does not satisfy security requirements because we are not allowed to have any third party with access to encryption keys. What I want to know is whether Vormetric's transparent data encryption works for S3, and if so, is it truly transparent, i.e., everything is the same from the user's point of view with it or without it.
@Peter Coates Check out this page for Vormetrics. I have not used them before but it looks like they have an agent or appliance that encrypts data for you. With hive you should check out the benchmark and make sure it has enough throughput in and out of S3.
Regarding your question: What I want to know is whether Vormetric's transparent data encryption works for S3, and if so, is it truly transparent, i.e., everything is the same from the user's point of view with it or without it.
Vormetric’s Cloud Encryption Gateway released in Q2 2015 does exactly that: https://www.vormetric.com/company/newsroom/press-releases/vormetric-announces-cloud-encryption-gatew...
Regarding your question (critical to enterprise implementations especially in highly regulated industries): Is there another solution that maintains keys entirely under the user's control?
The Cloud Encryption Gateway gives the owner full control over the keys as well. Accessing their gateway from your own proxy server takes this a step further.
The above link has full details but relevant text from it are:
This new product extends Vormetric’s award winning data security platform with protection for data in cloud storage environments, helping enterprises secure sensitive data residing in Amazon Web Services (AWS) Simple Storage Service (S3) and Box environments with encryption, key management and access controls.
With the Vormetric Cloud Encryption Gateway, data is encrypted before it is saved to cloud storage, while encryption keys and access policies are always under enterprise control. The solution consists of two major components - the gateway provides encryption and policy enforcement, and is paired with a Vormetric Data Security Manager (DSM) for encryption key and policy management. Both are available as virtual appliances, and the DSM may also be deployed as a FIPS 140-2 Level 2 or Level 3 certified hardware appliance. The combined solution removes the possibility of encryption key or data compromise at the cloud storage vendor location, while enabling security teams to establish the visibility and control required to keep assets secure and meet compliance requirements.
There may be other proprietary solutions as well ... this answer speaks specifically to Vormetric.