question Re: Ranger KMS not Decrypting Files in Archives of Support Questions (Read Only)

Ranger KMS not Decrypting Files

nicholas_pilegg — Thu, 08 Dec 2016 22:11:53 GMT

Hello,

I have been trying to pull some data from our SQL Server into hdfs via sqoop. The destination point is an encrypted zone (/secure/). The files are written and when I pull the files with hdfs dfs -get /secure/[folder imported] I am getting gibberish when I open the files. My first though was I couldn't decrypt the file, but when I look at the audit logs in Ranger, I am seeing the access type decrypteek for my user on the read and the write. Below is the sqoop query. Any insights would be great.

sqoop import \

-D sqoop.test.import.rootDir=hdfs://popul/secure/ \

--target-dir hdfs://popul/secure/intest/ \

--connect "jdbc:sqlserver://[serverip]:1433;database=[database]" \

--username [sqoopuser] \

--password [password] \

--table S_Elg \

--fields-terminated-by "|" \

--columns "col1, col2, col3" \

--split-by ElgKey \

-- --schema ACC

P.S. when I run this query in a non encrypted zone, everything works as expected.

Nick

Re: Ranger KMS not Decrypting Files

RyanCicak — Sun, 18 Dec 2016 04:14:26 GMT

Hi @Nick Pileggi

Can you verify the user trying to read the file has the Decrypt EEK permission in Ranger KMS? You can use my article here as a reference

Is your cluster Kerberized?

Re: Ranger KMS not Decrypting Files

nicholas_pilegg — Wed, 21 Dec 2016 22:47:43 GMT

@Ryan Cicak

After staring at this with a hortonworks engineer (who was onsite for an unrelated reason), we figured out the problem. The whole time Ranger KMS was doing its job, but I had enabled compression on my mapper outputs with these changes:

mapreduce.map.output.compress	true
mapreduce.output.fileoutputformat.compress	true

When I pulled the outputs of my sqoop job they looked like binary, but in reality they were just compressed. After deflating them everything is working correctly.

Nick