Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Issue while configuring Encryption (SSE-KMS) for Amazon S3

avatar
Expert Contributor

We are facing issues while trying to configure Hadoop cluster to use Server Side Encryption on Amazon S3.

Error Message:

Error: java.lang.RuntimeException: Hive Runtime Error while closing operators: org.apache.hadoop.fs.s3a.AWSS3IOException: saving output on XXX.db/.hive-staging_hive_2017-06-02_05-35-09_761_2361068000315861367-1/_task_tmp.-ext-10001/_tmp.000000_3: com.amazonaws.services.s3.model.AmazonS3Exception: The encryption method specified is not supported (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: DBE44B5B11EA4CAC), S3 Extended Request ID: 0S6KxrK+bNFU0uZAZVju1EpL1QIGH4bvmbffWNxb/BR3r5540ZWQ47Qtk/L/WIWReO239mfkxYM=: The encryption method specified is not supported (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: DBE44B5B11EA4CAC)

HDP Version-2.5.3.0

Ambari Version-2.4.2.0

Hive-1.2.1

https://issues.apache.org/jira/browse/HADOOP-13075.

Does hdp support SSE-KMS? If yes, then what are the required configuration?

Note: We can use SSE-S3 Encryption (AES256) without any issues.

Added below property in core-site.xml to use SSE-KMS.

<property>
    <name>fs.s3a.server-side-encryption-algorithm</name>
    <value>SSE-KMS</value>
</property>
<property>
    <name>fs.s3a.server-side-encryption-key</name>
    <value>your_kms_key_id</value>
</property>
1 ACCEPTED SOLUTION

avatar

@Shyam Shaw - As @Dominika Bialek also mentioned, this feature was introduced in HDP 2.6.1

I have confirmed this with HDFS Development team and this is not part of HDP 2.5.3

Please upgrade your cluster to the latest HDP version (HDP 2.6.1) to use this feature.

Hope this clarifies your issue.

View solution in original post

3 REPLIES 3

avatar

@Shyam Shaw

I know that we support KMS in HDP 2.6.1. Required configuration is described at https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_cloud-data-access/content/SSE-KMS.html.

I am not sure about HDP 2.5.3. Maybe @stevel can answer this.

avatar

@Shyam Shaw - As @Dominika Bialek also mentioned, this feature was introduced in HDP 2.6.1

I have confirmed this with HDFS Development team and this is not part of HDP 2.5.3

Please upgrade your cluster to the latest HDP version (HDP 2.6.1) to use this feature.

Hope this clarifies your issue.

avatar
Expert Contributor

@Namit Maheshwari @Dominika Bialek

Thanks for your response.