Data Encryption and Decryption across the source and Target folders

[#1]

Hi I am new to Hadoop Platform and i have been asked to work on the data encryption and decryption.

Let me give you the background so that you can suggest me how to achieve this task. - I am sqooping the data from Teradata to my HDFS stg/src-a folder and creating external table on stg/src-a and pushing this data to my another layer tgt/input - this one is managed table, once data populated from stg/src-a to tgt/input, will cleanup stg/src-a folder.

whenever i bring data to stg/src-a, i need to left outer join with tgt/input data to load only newer record and wherever there is a match update to be triggered based on certain conditions. - question is, when i encrypt tgt/input folder it is not allowing to read data to join with stg/src-a external table.

Question 2: if the above one is successful, i need to expose the data to end-user by splitting total table and for some of the user i need to expose portion of the table by creating views or so. - If anyone of you worked on similar activites, please show some light on that. Thanks

[#2]

Are you using Transparent Data Encryption on HDFS?

[#3]

Yes Deepesh.

[#4 · Accepted solution]

Are you using Hive? The only support engine for Hive to run with HDFS TDE is MR. Hive on Tez doesn't work yet today but will be in the near future. If you are using Hive on MR, can you post here the exception? What is your ranger policy defined for that encrypted folder?

[#5]

Yes I am using Hive not Hive on Tez - I am not sure about the policy what has been defined.

[#6]

Are you still having issues babu?