Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Transparent Data encryption (TDE) and Hive on Tez

Solved Go to solution

Transparent Data encryption (TDE) and Hive on Tez

New Contributor

Hi,

I am trying out using HDFS TDE along with Hive on Tez. I see in the documentation that TDE with Hive on Tez is not supported but am able to successfully create a table and select rows from it.

I am creating a data with encrypted data and then creating a table (external) pointing to the encrypted data. I am able to successfully select rows from this table. I am concerned that Hive on Tez is documented as not being supported. I was wondering what part of Hive on Tez is not supported? How is it possible that my test case works? Is it something to do with using external tables?

Thanks.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Transparent Data encryption (TDE) and Hive on Tez

@Anil Mathew

Joining the conversation late but maybe can shed some light.

As far as TDE with Tez is concerned, it works and is supported with HDP 2.4. The caveat though is that the intermediary "shuffle" data will not be encrypted if it overflows to disk. This is temporary, short lived data though, if it does overflow to disk, and may not be an issue depending on your requirements.

TDE and Spark should also be supported with HDP 2.5 but will also have the same caveat.

3 REPLIES 3

Re: Transparent Data encryption (TDE) and Hive on Tez

@Anil Mathew

Hi Anil, You have answered your own question. You can select the rows but there is chance that more complex queries may not work, data format or consistency can be an issue "unofficial statement" .

Also, Have you tired locally managed tables on TDE with orc and compression? That may cause unknown issues.

The following components do not currently support HDFS data at rest encryption:

  • Hive on Tez
  • Spark
  • HDP Search
  • Storm
  • Accumulo
  • Falcon

Re: Transparent Data encryption (TDE) and Hive on Tez

@Anil Mathew

Joining the conversation late but maybe can shed some light.

As far as TDE with Tez is concerned, it works and is supported with HDP 2.4. The caveat though is that the intermediary "shuffle" data will not be encrypted if it overflows to disk. This is temporary, short lived data though, if it does overflow to disk, and may not be an issue depending on your requirements.

TDE and Spark should also be supported with HDP 2.5 but will also have the same caveat.

Re: Transparent Data encryption (TDE) and Hive on Tez

New Contributor

Please refer to the official HDP2.4 docs here:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_Security_Guide/content/configuring-encry...

Currently, the following are supported in HDP2.4 for TDE:

  • Hive
  • Hive on Tez
  • HBase
  • Sqoop
  • YARN
  • MapReduce
  • Oozie
  • WebHDFS
Don't have an account?
Coming from Hortonworks? Activate your account here