Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HBase snapshots as Map-reduce job input - permission denied

Solved Go to solution

Re: HBase snapshots as Map-reduce job input - permission denied

Contributor

Ok. so here is the complete situation.

When you run a MR on top of a Snapshot, the MR framework will look at all the inputs and create all the tasks for that. However, those tasks might have to wait for some time to be executed depending on the number of slots available on the cluster vs the number of tasks.

The issue is, if while the tasks are pending one of the input is move/deleted/split/merged, etc. then the splits are not pointing anymore to a valid input and the MR job wil fail.

To avoid that, we have to create al lthe links to all the inputs to make sure HBase keep a reference to those files even if they have to me moved, the same way a snapshot is doing. The issue is, those links have to be in the /hbase folder. And this is why you need the rights for that.

So to be able to run a MR job on top of a snapshot you need a user with reads/writes access to the /hbase folder. This should be fixed in HBase 1.2 (but it's just on the plans for now and you will need to double check wen we will be closer to 1.2).

Also, please keep in mind that doing MR on top of Snapshots bypass all the HBase layers. Therefore, if there is any ACLs or Cell level security activated on the initial table, then will all by bypassed by the MR job. Everything will be readable by the job.

Let me know if you have any other question or if I can help with anything.

HTH.

JMS

View solution in original post

Highlighted

Re: HBase snapshots as Map-reduce job input - permission denied

Hi Jean-Marc,

Thanks for your thorough analysis!
Making sure the HFiles stay around makes perfect sense, so it is just a permissions issue.
And hopefully this will be fixed with HBase 1.2 then ? I will use a permissions workaround meanwhile.

Best regards
Jost


Don't have an account?
Coming from Hortonworks? Activate your account here