Created 06-02-2017 12:25 PM
We are facing issues while trying to configure Hadoop cluster to use Server Side Encryption on Amazon S3.
Error Message:
Error: java.lang.RuntimeException: Hive Runtime Error while closing operators: org.apache.hadoop.fs.s3a.AWSS3IOException: saving output on XXX.db/.hive-staging_hive_2017-06-02_05-35-09_761_2361068000315861367-1/_task_tmp.-ext-10001/_tmp.000000_3: com.amazonaws.services.s3.model.AmazonS3Exception: The encryption method specified is not supported (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: DBE44B5B11EA4CAC), S3 Extended Request ID: 0S6KxrK+bNFU0uZAZVju1EpL1QIGH4bvmbffWNxb/BR3r5540ZWQ47Qtk/L/WIWReO239mfkxYM=: The encryption method specified is not supported (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: DBE44B5B11EA4CAC)
HDP Version-2.5.3.0
Ambari Version-2.4.2.0
Hive-1.2.1
https://issues.apache.org/jira/browse/HADOOP-13075.
Does hdp support SSE-KMS? If yes, then what are the required configuration?
Note: We can use SSE-S3 Encryption (AES256) without any issues.
Added below property in core-site.xml to use SSE-KMS.
<property> <name>fs.s3a.server-side-encryption-algorithm</name> <value>SSE-KMS</value> </property> <property> <name>fs.s3a.server-side-encryption-key</name> <value>your_kms_key_id</value> </property>
Created 06-02-2017 09:33 PM
@Shyam Shaw - As @Dominika Bialek also mentioned, this feature was introduced in HDP 2.6.1
I have confirmed this with HDFS Development team and this is not part of HDP 2.5.3
Please upgrade your cluster to the latest HDP version (HDP 2.6.1) to use this feature.
Hope this clarifies your issue.
Created 06-02-2017 06:07 PM
I know that we support KMS in HDP 2.6.1. Required configuration is described at https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_cloud-data-access/content/SSE-KMS.html.
I am not sure about HDP 2.5.3. Maybe @stevel can answer this.
Created 06-02-2017 09:33 PM
@Shyam Shaw - As @Dominika Bialek also mentioned, this feature was introduced in HDP 2.6.1
I have confirmed this with HDFS Development team and this is not part of HDP 2.5.3
Please upgrade your cluster to the latest HDP version (HDP 2.6.1) to use this feature.
Hope this clarifies your issue.
Created 06-04-2017 08:57 AM
@Namit Maheshwari @Dominika Bialek
Thanks for your response.