Transparent Data encryption (TDE) and Hive on Tez

[#1]

Hi,

I am trying out using HDFS TDE along with Hive on Tez. I see in the documentation that TDE with Hive on Tez is not supported but am able to successfully create a table and select rows from it.

I am creating a data with encrypted data and then creating a table (external) pointing to the encrypted data. I am able to successfully select rows from this table. I am concerned that Hive on Tez is documented as not being supported. I was wondering what part of Hive on Tez is not supported? How is it possible that my test case works? Is it something to do with using external tables?

Thanks.

[#2]

@Anil Mathew

Hi Anil, You have answered your own question. You can select the rows but there is chance that more complex queries may not work, data format or consistency can be an issue "unofficial statement" .

Also, Have you tired locally managed tables on TDE with orc and compression? That may cause unknown issues.

The following components do not currently support HDFS data at rest encryption:

• Hive on Tez
• Spark
• HDP Search
• Storm
• Accumulo
• Falcon

[#3 · Accepted solution]

@Anil Mathew

Joining the conversation late but maybe can shed some light.

As far as TDE with Tez is concerned, it works and is supported with HDP 2.4. The caveat though is that the intermediary "shuffle" data will not be encrypted if it overflows to disk. This is temporary, short lived data though, if it does overflow to disk, and may not be an issue depending on your requirements.

TDE and Spark should also be supported with HDP 2.5 but will also have the same caveat.

[#4]

Please refer to the official HDP2.4 docs here:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_Security_Guide/content/configuring-encry...

Currently, the following are supported in HDP2.4 for TDE:

• Hive
• Hive on Tez
• HBase
• Sqoop
• YARN
• MapReduce
• Oozie
• WebHDFS