HBase Insufficient Permissions with Kerberos

[#1]

I'm trying to set up the ACLs for HBase now that we have enabled secure authentication. I'm using the HBase shell, but I don't have permissions to grant anything:

ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions (user=<redacted>, scope=GLOBAL, action=ADMIN)

I have added my user name to the "HBase Superuser" setting under configuration and re-deployed the service, but the error persists.

[#2]

Could you share your CDH version? I'm unable to reproduce this with a
username added (without @ character prefix) to the config you've mentioned
in the recent CDH 6.x releases.

By 're-deployed' did you mean restart? I had to restart the service for all
hosts to see the new superuser config.

[#3]

How are you using the user to group resolution? Have you added your user name on all the nodes? Or are you using LDAP/AD integration? Because it can be that the Hbase node does not know that you are a member of hbase supersuser group

[#4]

I am using CDH 5.15.0; I did a rolling restart.

We are using a centralized AD to authenticate and store the Kerberos principals. We don't have an AD group specifically for HBase, although I'm a member of the admin and ETL groups. Is there somewhere I need to configure HBase with a superuser group? I did add my account as an HBase Superuser, but it didn't resolve the issue.

[#5]

Thank you for confirming the details,

Does the subject part of your klist output match the added username in the HBase Superusers configuration precisely?

If your user is in a different realm than the cluster services, is the realm name present as part of HDFS -> Configuration -> 'Trusted Realms'?

Are all commands done as the superuser failing? What HBase shell command/operation specifically is leading to your quoted error?

As to adding groups, it can be done in the same field, except you need to add an '@' prefix to the name. For ex. if your group is cluster_administrators, then add it in as '@cluster_administrators' in the HBase Superusers config. When using usernames, the @ must not be specified. Both approaches should work though.

P.s. If you'll be relying on groups, ensure all cluster hosts return consistent group lookup output for id <user> commands, as the authorization check is distributed across the cluster roles for HBase.

[#6]

Yes, the klist output matches the added username, and everything is in the same realm.

I'm trying to just create a table with the HBase shell:

hbase(main):002.0>create 'testtable', 'colfam1'

This results in the error from my first post:

ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions (user=<username>@<realm>, scope=default, params=[namespace=default, table=default:testtable,family=colfam1],action=CREATE)

[#7]

Ok, I figured it out. There was a mapping rule that translated my Kerberos principal name to a lower-case short name, i.e.

USER1@EXAMPLE.COM becomes user1

I had entered both USER1 and USER1@EXAMPLE.COM as HBase superusers, but not user1. Tricky. . .