User impersonation fails in secured HBase

[#1]

Using HDP 3.0.0 bundle which has Hadoop 3.1.0 and HBase 2.0.0.3.0 (Version 2, Alpha release-3; Why??). Ambari is used to manage the setup.

We are trying to achieve user impersonation using Rest proxy in an unsecured and then secured setup. Tried it via code (UGI and doAs() as here, https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html, but for HBase). All of these fail and return the data, which has visibility tags attached, for the super user (hbase). That is, all data.

Rest proxy path: curl -> Knox -> HBase REST Gateway -> HBase

In a secured setup, Knox authenticates as knox principal to Gateway, Gateway authenticates to HBase as 'hbase' (tried 'HTTP' user as well) and HBase authenticates as 'hbase'.

The users to be impersonated are ldap authenticated at Knox using the Shiro authentication provider and a default identity assertion provider maps the logged in user to the ?doAs query param passed in the call to HBase Rest Gateway.

The logged in users have 'R' permission granted to the table being queried. One user has auth set for the visibility tags against some fields. Second user is not authorized against the visibility tags.

However, all data is returned from the table always as if the superuser is executing the table query, not the real user impersonated by the superuser as expected.

The required config properties for hadoop.proxyuser and the hbase.security (both authn and authz) are specified in the hbase site xml (tried on the HDFS core site xml as well). Config is matching as specified here -> https://community.cloudera.com/t5/Support-Questions/Hbase-rest-proxying/m-p/150395

Towards the end, i've just left this config in: hbase.rest.support.proxyuser: true. And deleted the rest of config that is required to satisfy the proxy auth check for the superuser. Got no errors. Worked merrily as before. Similarly, i've left half config in and removed others hoping the proxy auth check would fail. Again no effect.

On my eyeballs widening on seeing HBase version 2, Alpha Rel-3 packaged in HDP 3.0.0, i've narrowed down to it being the culprit. I've proposed installing HDP 3.1.0 which has HDFS 3.1.1 and HBase 2.0.2. Will take it for a spin once sys admin spit it out.

My question is why did HDP 3.0.0 bake in an Alpha release of HBase?

Secondly, what else should i look at to make user impersonation work (remember i've tried impersonation via Rest path as well as code in secured as well as unsecured Hadoop)?

[#2]

Ok. Upgraded to HDP 3.1.0 (HDFS 3.1.1 and HBase 2.0.2). Unsecured setup. Started up HBase Rest daemon (as hbase user) and sent a curl request with ?doAs=realuser. Query returns all data as if executed by 'hbase' superuser as opposed to 'realuser' who does not have authorization to visibility tags attached to some fields.

The required config (''hbase.rest.support.proxyuser=true et al as specified in original post) was specified via Ambari on the HBase custom site xml.

What am i doing wrong??

[#3]

Make sure you have changed these properties in your hbase-site.xml:

<property>
<name>hbase.thrift.support.proxyuser</name>
<value>true</value>
</property>

<property>
<name>hbase.regionserver.thrift.http</name>
<value>true</value>
</property>

Then check in HDFS-->Configs-->Advanced-Custome core-site.xml add these 2 properties to enable HBase to impersonates a user

 <property>
<name>hadoop.proxyuser.hbase.hosts</name>
<value>*</value>
</property>

<property>
<name>hadoop.proxyuser.hbase.groups</name>
<value>*</value>
</property>                       

Please do these modifications and start any stale configs and revert

[#4]

Hi @Shelton Thanks for your prompt reply.

We tried your suggested changes and it worked! BUT only when the Thrift server is running.

Impersonation in a HBase REST setup does not work for us. Essentially, with all the required config and testing in secured as well as unsecured Hadoop and across different versions as above, we have come to the conclusion that the REST doAs functionality is buggy.

[#5]

I'm having a very similar issue.

I posted this on the 08-08-2019

https://community.cloudera.com/t5/Support-Questions/Ranger-hbase-plugin-not-proxying-users-Runs-ever...

HDP 3.0.1 hbase 2.0.0.3.0.1.0-187

Knox is dispatching the request to hbase correctly.

19/09/05 14:39:31 ||c5231017-9998-458d-9336-98721bcb7cb2|audit|39.7.48.21|WEBHBASE|c8ary|||dispatch|uri|https://<namenode>:60080/footy/1?doAs=andrew|unavailable|Request method: GET
19/09/05 14:39:31 ||c5231017-9998-458d-9336-98721bcb7cb2|audit|39.7.48.21|WEBHBASE|c8ary|||dispatch|uri|https://<namenode>:60080/footy/1?doAs=andrew|success|Response status: 200
19/09/05 14:39:31 |||audit|39.7.48.21|WEBHBASE|andrew|||access|uri|/gateway/default/hbase/footy/1|success|Response status: 200

Every request is run as HTTP (which I have specified as the rest principal and rest authentication principal)

hbase.rest.authentication.kerberos.keytab

/etc/security/keytabs/spnego.service.keytab

hbase.rest.authentication.kerberos.principal

HTTP/_HOST@REALM

hbase.rest.kerberos.principal

HTTP/_HOST@REALM

hbase.rest.keytab.file

/etc/security/keytabs/spnego.service.keytab

hbase-ranger-plugin is working. When I remove the HTTP user from the ranger policy access is denied. 

doAs functionality seems buggy to me also.

[#6]

Hi @andrew_ryan1 

as I suspected, suffering is universal in a distributed world...

Please ensure you re-start HBase Rest daemon after every config change:

./hbase-daemon.sh [stop|start] rest --port <8090>

This, i suspect, is because the Rest daemon refers to the Hbase config. Re-starting affected servers after a config change from Ambari unfortunately leaves out the Rest daemon.

As for your problem in the linked post in your reply, when you say all actions are run as root, I'm assuming you mean 'hbase' user? If you literally mean 'root', then it could be the Linux 'root' user may have been used to start the Rest daemon. This happened at my joint 😉

[#7]

Hi @hbased

Did you get it working?

Yeah, I was running the rest server as root with no Kerberos enabled. Curl requests through Knox were all run as root.

Now with Kerberos enabled and running the rest server as hbase rather than root every request is run as HTTP. Which is my proxyuser. 

I have been restarting the rest server but will try again tomorrow as a sanity check.

Cheers

[#8]

Hi @andrew_ryan1 

We gave up on Rest but got proxying working through Thrift (using tHttp Client, then tclient.setCustomHeader("doAs", "realuser") where realuser is the user that the superuser will be proxying in for).

We are now scaling the hurdle of getting impersonation working with HBase cell-based security authorization off real user's group memberships.

You might also want to try adding these props for the HTTP user to the HDFS core-site.xml:

hadoop.proxyuser.HTTP.hosts=*

hadoop.proxyuser.HTTP.groups=*

hadoop.proxyuser.HTTP.users=*

i have looked at the HBase proxy authorization code and it simply looks for these config props for the logged in user (if 'hbase' user, hadoop.proxyuser.hbase.<> props must exist) and match the values ('*' above) against the values that came in on the client call. Kerberos is not a pre-requisite for this to work.

[#9]

Hi,

Did you also checked in the Knox configuration if the port for WEBHBASE is matching the port from hbase rest?

[#10]

Hi @hbased 

I restarted my hbase rest servers - no joy.

I have the hadoop.proxyuser.HTTP.<> settings in core site.

I think I'll have a trawl through the hbase code and raise a jira.

Good luck with the cell based auth.

Thanks for your help!

Cheers

Andrew

[#11]

Hi @andrew_ryan1 we got impersonation working using Thrift for users (based on either their user id or groups) when we added property below to HBase site xml:

<property>
  <name>hbase.security.exec.permission.checks</name>
  <value>true</value>
</property>

This should work for the REST route as well as it is not specific to REST or Thrift. Needless to say, the real user or their groups need to be given the appropriate privileges (ACLs / visibility tag authorizations).

Additionally, these properties added to core-site.xml would enable group lookup for a real user, so groups needed for authorizing a real user can be enumerated (we are yet to test this):

<property>

  <name>hadoop.security.group.mapping</name>

  <value>org.apache.hadoop.security.LdapGroupsMapping</value>

</property>

<property>

  <name>hadoop.security.group.mapping.ldap.url</name>

  <value>ldap://server</value>

</property>

<property>

  <name>hadoop.security.group.mapping.ldap.bind.user</name>

  <value>Administrator@example-ad.local</value>

</property>

<property>

  <name>hadoop.security.group.mapping.ldap.bind.password</name>

  <value>****</value>

</property>

<property>

  <name>hadoop.security.group.mapping.ldap.base</name>

  <value>dc=example-ad,dc=local</value>

</property>

<property>

  <name>hadoop.security.group.mapping.ldap.search.filter.user</name>

  <value>(&amp;(objectClass=user)(sAMAccountName={0}))</value>

</property>

<property>

  <name>hadoop.security.group.mapping.ldap.search.filter.group</name>

  <value>(objectClass=group)</value>

</property>

<property>

  <name>hadoop.security.group.mapping.ldap.search.attr.member</name>

  <value>member</value>

</property>

<property>

  <name>hadoop.security.group.mapping.ldap.search.attr.group.name</name>

  <value>cn</value>

</property>

[#12]

Hi @hbased;

It is a bug. There is a jira for the issue and apparently it was resolved in version 3.0.0. This is the apache hbase site - not necessarily your distro version. Cloudera say the issue is resolved in 3.1.4. We have raised a support ticket with Cloudera and they are patching our current distro version of hbase and providing us with a new binary.

Here's the jira ref.

https://issues.apache.org/jira/browse/HBASE-21960

All the best