HBase Authorization Issue

[#1]

I have a cloudera cluster up and running. I have Knox which forwards requests to webhbase and hbase uses ranger for authorization. Ranger is connected to FreeIPA LDAP and we use kerberos internally for authentication.

In Ranger, I have a policy that gives access to read a table. In that same ranger policy a group in my freeIPA instance has the ability to access that table. I am having an issue. When I remove a member from that group in freeIPA and rerun ranger user sync, I try a curl call to get data from the table it still works. However when I run it again it gives the expected result as a denial. This has been a consistent pattern for all changes to user membership in ldap groups. 

The Ranger HBase Policy Sync is working as expected and the Usersync is working as expected as well. After I run Usersync, I confirm that the user has been removed from that group in Ranger but it's still allowing me access. 

Does anyone know why this is? There are similar Kafka and HDFS policies where I try access those resources as well using the same group and they work the first time but for HBase, it's taking two calls for it to work correctly.

Any help would be greatly appreciated!

[#2]

@drewski7 

Review the Ranger plugin configuration for HBase to understand its caching settings. Look for properties related to caching and cache refresh intervals. You can find these settings in the ranger-hbase-security.xml configuration file or in the Ranger Admin UI under the HBase repository configuration.

Also Try manually refreshing the Ranger policies in the Ranger Admin UI after running the user sync. This might help in invalidating any stale cache entries.

Check the HBase logs for any messages related to authentication and authorization. Look for any log entries that might indicate caching behavior or delays in applying new policies.

If you identify caching settings related to the TTL (Time-To-Live) for cached entries, consider reducing this value to ensure that changes in group memberships are picked up more quickly.
Verify that the Kerberos ticket cache is being refreshed properly. Sometimes, stale Kerberos tickets might cause inconsistencies in access control.

ranger.plugin.hbase.policy.pollIntervalMs: This setting controls how often the Ranger plugin polls for policy changes. Lowering this value might help in picking up changes more quickly.
ranger.plugin.hbase.authorization.cache.max.size: This setting controls the maximum size of the authorization cache. Adjusting this might help if the cache is too large and not being refreshed adequately.

Check hbase.security.authorization and hbase.security.authentication settings in hbase-site.xml to ensure they are configured correctly.

[#3]

@RAGHUY 

Thank you for your response. I tried manually resetting the policies and also resetting the cache for hbase but still getting the same thing. I am not making changes to any hbase policies because that is using a group for authorization. I am only making changes to ldap membership group in ldap. Then I rerun user sync and I see the user has been removed from that group in ranger admin UI. However I am still able to get acccess to the table the first time but the second time I can’t it works. The opposite happens too when I add a user to a group the first time it doesn’t work and the second time it works. I the HBase rest server logs I am seeing unauthorized and some issues with Kerberos thread unexpectedly exiting. Do you think that has something to do with it?

[#4 · Accepted solution]

@drewski7 

Ensure that the Kerberos tickets are being refreshed properly for the HBase REST server. Stale or expired tickets might cause intermittent authorization issues.
Check the Kerberos cache to ensure that it is being updated correctly when group memberships change in LDAP.
Restart the HBase REST server after making changes to the LDAP group and running the user sync to see if it resolves the inconsistency.
Analyze the HBase REST server logs more thoroughly, especially the messages related to unauthorized access and Kerberos thread issues. Look for patterns or specific errors that could provide more clues.
Verify the settings for ranger.plugin.hbase.policy.pollIntervalMs and ranger.plugin.hbase.authorization.cache.max.size again, and experiment with lowering the poll interval to see if it improves the responsiveness of policy changes.
In the Ranger Admin UI, after running the user sync, manually refresh the policies for HBase and observe if this action has any immediate effect on the authorization behavior.
Confirm that there are no discrepancies in the policies displayed in the Ranger Admin UI and the actual enforcement in HBase.
Double-check the synchronization between FreeIPA LDAP and Ranger. Ensure that the user sync is not just updating the Ranger Admin UI but is also effectively communicating changes to all Ranger plugins.
Review the user sync logs to verify that all changes are processed correctly without errors.