Issue while configuring Encryption (SSE-KMS) for Amazon S3

[#1]

We are facing issues while trying to configure Hadoop cluster to use Server Side Encryption on Amazon S3.

Error Message:

Error: java.lang.RuntimeException: Hive Runtime Error while closing operators: org.apache.hadoop.fs.s3a.AWSS3IOException: saving output on XXX.db/.hive-staging_hive_2017-06-02_05-35-09_761_2361068000315861367-1/_task_tmp.-ext-10001/_tmp.000000_3: com.amazonaws.services.s3.model.AmazonS3Exception: The encryption method specified is not supported (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: DBE44B5B11EA4CAC), S3 Extended Request ID: 0S6KxrK+bNFU0uZAZVju1EpL1QIGH4bvmbffWNxb/BR3r5540ZWQ47Qtk/L/WIWReO239mfkxYM=: The encryption method specified is not supported (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: DBE44B5B11EA4CAC)

HDP Version-2.5.3.0

Ambari Version-2.4.2.0

Hive-1.2.1

https://issues.apache.org/jira/browse/HADOOP-13075.

Does hdp support SSE-KMS? If yes, then what are the required configuration?

Note: We can use SSE-S3 Encryption (AES256) without any issues.

Added below property in core-site.xml to use SSE-KMS.

<property>
    <name>fs.s3a.server-side-encryption-algorithm</name>
    <value>SSE-KMS</value>
</property>
<property>
    <name>fs.s3a.server-side-encryption-key</name>
    <value>your_kms_key_id</value>
</property>

[#2]

@Shyam Shaw

I know that we support KMS in HDP 2.6.1. Required configuration is described at https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_cloud-data-access/content/SSE-KMS.html.

I am not sure about HDP 2.5.3. Maybe @stevel can answer this.

[#3 · Accepted solution]

@Shyam Shaw - As @Dominika Bialek also mentioned, this feature was introduced in HDP 2.6.1

I have confirmed this with HDFS Development team and this is not part of HDP 2.5.3

Please upgrade your cluster to the latest HDP version (HDP 2.6.1) to use this feature.

Hope this clarifies your issue.

[#4]

@Namit Maheshwari @Dominika Bialek

Thanks for your response.