Support Questions
Find answers, ask questions, and share your expertise

Issue while configuring Encryption (SSE-KMS) for Amazon S3

Solved Go to solution
Highlighted

Issue while configuring Encryption (SSE-KMS) for Amazon S3

Expert Contributor

We are facing issues while trying to configure Hadoop cluster to use Server Side Encryption on Amazon S3.

Error Message:

Error: java.lang.RuntimeException: Hive Runtime Error while closing operators: org.apache.hadoop.fs.s3a.AWSS3IOException: saving output on XXX.db/.hive-staging_hive_2017-06-02_05-35-09_761_2361068000315861367-1/_task_tmp.-ext-10001/_tmp.000000_3: com.amazonaws.services.s3.model.AmazonS3Exception: The encryption method specified is not supported (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: DBE44B5B11EA4CAC), S3 Extended Request ID: 0S6KxrK+bNFU0uZAZVju1EpL1QIGH4bvmbffWNxb/BR3r5540ZWQ47Qtk/L/WIWReO239mfkxYM=: The encryption method specified is not supported (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: DBE44B5B11EA4CAC)

HDP Version-2.5.3.0

Ambari Version-2.4.2.0

Hive-1.2.1

https://issues.apache.org/jira/browse/HADOOP-13075.

Does hdp support SSE-KMS? If yes, then what are the required configuration?

Note: We can use SSE-S3 Encryption (AES256) without any issues.

Added below property in core-site.xml to use SSE-KMS.

<property>
    <name>fs.s3a.server-side-encryption-algorithm</name>
    <value>SSE-KMS</value>
</property>
<property>
    <name>fs.s3a.server-side-encryption-key</name>
    <value>your_kms_key_id</value>
</property>
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Issue while configuring Encryption (SSE-KMS) for Amazon S3

@Shyam Shaw - As @Dominika Bialek also mentioned, this feature was introduced in HDP 2.6.1

I have confirmed this with HDFS Development team and this is not part of HDP 2.5.3

Please upgrade your cluster to the latest HDP version (HDP 2.6.1) to use this feature.

Hope this clarifies your issue.

View solution in original post

3 REPLIES 3
Highlighted

Re: Issue while configuring Encryption (SSE-KMS) for Amazon S3

@Shyam Shaw

I know that we support KMS in HDP 2.6.1. Required configuration is described at https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_cloud-data-access/content/SSE-KMS.html.

I am not sure about HDP 2.5.3. Maybe @stevel can answer this.

Highlighted

Re: Issue while configuring Encryption (SSE-KMS) for Amazon S3

@Shyam Shaw - As @Dominika Bialek also mentioned, this feature was introduced in HDP 2.6.1

I have confirmed this with HDFS Development team and this is not part of HDP 2.5.3

Please upgrade your cluster to the latest HDP version (HDP 2.6.1) to use this feature.

Hope this clarifies your issue.

View solution in original post

Highlighted

Re: Issue while configuring Encryption (SSE-KMS) for Amazon S3

Expert Contributor

@Namit Maheshwari @Dominika Bialek

Thanks for your response.