Ranger KMS functionality behavior

[#1]

Hi team,

I have couple of questions on functionality.

What I have expected to using Ranger KMS is when the data is written in encrypted zone, the data should be in human readable but as below:

$ hdfs dfs -get /data/protegrity/data4.dat ./encrypted_data4.dat 

$ cat encrypted_data4.dat 
1AY&SX—“#„bd3ƒ'•  DE_ENC256®XQy”ª8@¿UuaùfšÆe4@ãoNVÕh¡}69þC$8¤ÌªÒÓ»Ö]\GR®´éXûš™?âëD
}‹]ê~+¨ÑN•Ä²z?iÄÝ 5ùDüt.ïÆ,+í/–öõZ9õXÙ+]R_#Ä×â6> 
¦KÂœÌ'„J   çÜÑâ,OzÝi.ځ^4WG­±´± 
2P‹qããE¼iåsLH'xH×oÚ6_ˆ'„ôE¦¯î©{_Hç˃ðîËíÒ†t¾+’:ÁÓ‡›°àå7¢@fH“9¾XTd/F'Îc9«þí  òûHýÁN‰QO4y5ànG¤wš2¢»<

Is this possible using Ranger KMS?

Secondly is it possible to do column level encryption in Hive/HBase using Ranger KMS?

Example as below:

0: jdbc:hive2://hortonworks.com> select * from table4;
+------------+---------------+---------------+-----------------------+------------------------+---------------------+
| table4.id  | table4.fname  | table4.lname  | table4.fake_prim_nss  | table4.fake_secnd_nss  | table4.fake_bod_dt  |
+------------+---------------+---------------+-----------------------+------------------------+---------------------+
| 1          | Sridhar       | Reddy         | 123456789             | 123456789              | 1990-03-23          |
| 2          | Happy         | Tom           | 234567890             | 234567890              | 1971-02-10          |
| 3          | Jun           | Yu            | 345678901             | 345678901              | 1972-10-23          |
+------------+---------------+---------------+-----------------------+------------------------+---------------------+
5 rows selected (0.255 seconds)
0: jdbc:hive2://hortonworks.com> select id, fname, lname, ptyProtectStr(cast(fake_prim_nss as string),'DE_nss23') as fake_prim_nss, fake_secnd_nss, fake_bod_dt, fake_bod_tms from table4;
+-----+---------+--------+----------------+-----------------+--------------+
| id  |  fname  | lname  | fake_prim_nss  | fake_secnd_nss  | fake_bod_dt  |
+-----+---------+--------+----------------+-----------------+--------------+
| 2   | Happy   | Tom    | 682585704      | 234567890       | 1971-02-10   |
| 1   | Sridhar | Reddy  | 115506653      | 123456789       | 1990-03-23   |
| 3   | Jun     | Yu     | 874950339      | 345678901       | 1972-10-23   |
+-----+---------+--------+----------------+-----------------+--------------+

Thirdly, how Ranger KMS will honor when you set hive doAs=false.

Any needful help is highly appreciated. Thanks in advance.

[#2 · Accepted solution]

1] Since KMS supports HDFS TDE (Transparent data encryption), client will decrypt the file during read so real content will be shown. If interested in seeing actual encrypted data, /.reserved/raw/<directory-path>/<filename> can be used.

2] Since the entire hive warehouse or hbase data dir is encrypted with HDFS TDE, column level encryption is not required.

3] If hive doAs is false, then hive user needs to be setup as proxy user in KMS.

[#3]

@vperiasamy

Awesome.

Can you please even let me know the permissions/functionality of below in Ranger KMS UI, it would be helpful if you can share any notes or links

Get 
Set Key Materials
Get Keys
Get Metadata

After installing Ranger KMS even though if the user is not having any permissions on location '/data/protegrity/' from Ranger, and having 'Decrypt EEK' permissions from Ranger KMS UI, user is able to read the data. My question is now, will the Ranger permissions(Read, Write, Create) will not honored on encrypted zone?

[#4]

@Sridhar Reddy - HDFS/Ranger permissions will continue to work as-is on encryption zone. If there are audit logs, please check how the user is getting read access to the folder (whether through Ranger ACL or Hadoop ACL).

Refer this link for KMS ACL.

[#5]

@vperiasamy

Thanks for confirm that Ranger ACL will work as-is. I will debug on it. Thanks for the help, you are the best... 🙂