Support Questions

Find answers, ask questions, and share your expertise

Data Encryption and Decryption across the source and Target folders

avatar
Explorer

Hi I am new to Hadoop Platform and i have been asked to work on the data encryption and decryption.

Let me give you the background so that you can suggest me how to achieve this task. - I am sqooping the data from Teradata to my HDFS stg/src-a folder and creating external table on stg/src-a and pushing this data to my another layer tgt/input - this one is managed table, once data populated from stg/src-a to tgt/input, will cleanup stg/src-a folder.

whenever i bring data to stg/src-a, i need to left outer join with tgt/input data to load only newer record and wherever there is a match update to be triggered based on certain conditions. - question is, when i encrypt tgt/input folder it is not allowing to read data to join with stg/src-a external table.

Question 2: if the above one is successful, i need to expose the data to end-user by splitting total table and for some of the user i need to expose portion of the table by creating views or so. - If anyone of you worked on similar activites, please show some light on that. Thanks

1 ACCEPTED SOLUTION

avatar
Expert Contributor

Are you using Hive? The only support engine for Hive to run with HDFS TDE is MR. Hive on Tez doesn't work yet today but will be in the near future. If you are using Hive on MR, can you post here the exception? What is your ranger policy defined for that encrypted folder?

View solution in original post

5 REPLIES 5

avatar

Are you using Transparent Data Encryption on HDFS?

avatar
Explorer

Yes Deepesh.

avatar
Expert Contributor

Are you using Hive? The only support engine for Hive to run with HDFS TDE is MR. Hive on Tez doesn't work yet today but will be in the near future. If you are using Hive on MR, can you post here the exception? What is your ranger policy defined for that encrypted folder?

avatar
Explorer

Yes I am using Hive not Hive on Tez - I am not sure about the policy what has been defined.

avatar
Expert Contributor

Are you still having issues babu?