Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Ranger KMS not Decrypting Files

avatar
Contributor

Hello,

I have been trying to pull some data from our SQL Server into hdfs via sqoop. The destination point is an encrypted zone (/secure/). The files are written and when I pull the files with hdfs dfs -get /secure/[folder imported] I am getting gibberish when I open the files. My first though was I couldn't decrypt the file, but when I look at the audit logs in Ranger, I am seeing the access type decrypteek for my user on the read and the write. Below is the sqoop query. Any insights would be great.

sqoop import \

-D sqoop.test.import.rootDir=hdfs://popul/secure/ \

--target-dir hdfs://popul/secure/intest/ \

--connect "jdbc:sqlserver://[serverip]:1433;database=[database]" \

--username [sqoopuser] \

--password [password] \

--table S_Elg \

--fields-terminated-by "|" \

--columns "col1, col2, col3" \

--split-by ElgKey \

-- --schema ACC

P.S. when I run this query in a non encrypted zone, everything works as expected.

Nick

1 ACCEPTED SOLUTION

avatar
Contributor

@Ryan Cicak

After staring at this with a hortonworks engineer (who was onsite for an unrelated reason), we figured out the problem. The whole time Ranger KMS was doing its job, but I had enabled compression on my mapper outputs with these changes:

mapreduce.map.output.compresstrue

mapreduce.output.fileoutputformat.compress

true

When I pulled the outputs of my sqoop job they looked like binary, but in reality they were just compressed. After deflating them everything is working correctly.

Nick

View solution in original post

2 REPLIES 2

avatar

Hi @Nick Pileggi

Can you verify the user trying to read the file has the Decrypt EEK permission in Ranger KMS? You can use my article here as a reference

Is your cluster Kerberized?

avatar
Contributor

@Ryan Cicak

After staring at this with a hortonworks engineer (who was onsite for an unrelated reason), we figured out the problem. The whole time Ranger KMS was doing its job, but I had enabled compression on my mapper outputs with these changes:

mapreduce.map.output.compresstrue

mapreduce.output.fileoutputformat.compress

true

When I pulled the outputs of my sqoop job they looked like binary, but in reality they were just compressed. After deflating them everything is working correctly.

Nick