question Re: Issue while configuring Encryption (SSE-KMS) for Amazon S3 in Archives of Support Questions (Read Only)

Issue while configuring Encryption (SSE-KMS) for Amazon S3

shyamshaw — Fri, 02 Jun 2017 19:25:59 GMT

We are facing issues while trying to configure Hadoop cluster to use Server Side Encryption on Amazon S3.

Error Message:

Error: java.lang.RuntimeException: Hive Runtime Error while closing operators: org.apache.hadoop.fs.s3a.AWSS3IOException: saving output on XXX.db/.hive-staging_hive_2017-06-02_05-35-09_761_2361068000315861367-1/_task_tmp.-ext-10001/_tmp.000000_3: com.amazonaws.services.s3.model.AmazonS3Exception: The encryption method specified is not supported (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: DBE44B5B11EA4CAC), S3 Extended Request ID: 0S6KxrK+bNFU0uZAZVju1EpL1QIGH4bvmbffWNxb/BR3r5540ZWQ47Qtk/L/WIWReO239mfkxYM=: The encryption method specified is not supported (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: DBE44B5B11EA4CAC)

HDP Version-2.5.3.0

Ambari Version-2.4.2.0

Hive-1.2.1

https://issues.apache.org/jira/browse/HADOOP-13075.

Does hdp support SSE-KMS? If yes, then what are the required configuration?

Note: We can use SSE-S3 Encryption (AES256) without any issues.

Added below property in core-site.xml to use SSE-KMS.

<property>
    <name>fs.s3a.server-side-encryption-algorithm</name>
    <value>SSE-KMS</value>
</property>
<property>
    <name>fs.s3a.server-side-encryption-key</name>
    <value>your_kms_key_id</value>
</property>

Re: Issue while configuring Encryption (SSE-KMS) for Amazon S3

Dominika — Sat, 03 Jun 2017 01:07:54 GMT

@Shyam Shaw

I know that we support KMS in HDP 2.6.1. Required configuration is described at https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_cloud-data-access/content/SSE-KMS.html.

I am not sure about HDP 2.5.3. Maybe @stevel can answer this.

Re: Issue while configuring Encryption (SSE-KMS) for Amazon S3

namaheshwari — Sat, 03 Jun 2017 04:33:11 GMT

@Shyam Shaw - As @Dominika Bialek also mentioned, this feature was introduced in HDP 2.6.1

I have confirmed this with HDFS Development team and this is not part of HDP 2.5.3

Please upgrade your cluster to the latest HDP version (HDP 2.6.1) to use this feature.

Hope this clarifies your issue.

Re: Issue while configuring Encryption (SSE-KMS) for Amazon S3

shyamshaw — Sun, 04 Jun 2017 15:57:46 GMT

@Namit Maheshwari @Dominika Bialek

Thanks for your response.