HBase Snapshotting Failed because of Failure to aquire SAS Token

[#1]

Hello,

<in a Public Cloud Environment in the Operational Database with Azure: Snapshots were failing.
The HBase-Service Command Tab showes that it occurs while copying a snapshot to another directory because it could not acquire a SAS token. (*)
Usually those errors are related to missing rights on Managed Identities. 
- But we double-checked all Azure Managed Identity IAMs and RBACs.
Sometimes the SAS-Token Failure is related to Kerberos which is disabled./>

What could have gone wrong? 

(*):
Client.RangerRESTClient: ===>> RangerRESTClient.init() : Since mKeyStoreType is NULL, setting System default.

[mKeyStoreType=jks] Exception in thread "main" Failed to acquire a SAS token for get-acl on / due to org.apache.hadoop.security.AccessControlException: Permission denied. at org.apache.hadoop.fs.azurebfs.services.AbfsClient.appendSASTokenToQuery(AbfsClient.java:1233)

[#2]

@MintberryCrunch, Welcome to our community! To help you get the best possible answer, I have tagged in our HBase/CM experts @9een @rki_ @SVB  who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#3]

@MintberryCrunch FYI

➤ In a Public Cloud Environment using Azure Operational Database (HBase), snapshot copy failures with a Permission denied error while acquiring a SAS token for get-acl on / typically stem from missing Access Control List (ACL) permissions on the root of the storage container, even if Azure Role-Based Access Control (RBAC) roles are correctly assigned.

➤ The AccessControlException at the root directory (/) indicates the driver is attempting to validate permissions at the top level before proceeding with the operation.

➤ Recommended Troubleshooting Steps
1. Grant the "Storage Blob Delegator" Role: Add this role to the Managed Identity used by the HBase service to ensure it can generate User Delegation SAS tokens.

2. Inspect ACLs via Storage Explorer: Use Azure Storage Explorer to right-click the root of the container and select Manage ACLs. Confirm the identity has at least Execute permissions.

3. Verify Firewall Settings: Confirm that "Allow trusted Microsoft services to access this storage account" is enabled in the Storage Account's Networking tab.

4. Check for Sticky Bits: Use the Azure CLI command az storage fs access show to see if the sticky bit is enabled on the target path.

[#4]

Hello 9een,

thank you for your reply!

1. How do I find out which role is used by the HBase Service? I used this guide to assign the roles ( https://docs.cloudera.com/cdp-public-cloud/cloud/requirements-azure/topics/mc-az-minimal-setup-for-… )
2. I assigned all identities all permissions to get something to work, because I was not sure which Managed Identity is responsible.
3. In my Azure Storage Account there is no specific "Allow trusted Microsoft services to access this storage Account" setting. But "Microsoft networking Routing" Routing preference is set. "Public network Access" is "Enabled form all networks".
4. All Sticky Bits are disabled on all path.

Regards.

[#5 · Accepted solution]

This Error refers to Hadoop ACLs
The Azure ACL and RBAC was correctly assigned.

The snapshot location must reside within the same directory as the HBase directory. The HBase user seems to have only access to this directory. For example, we used something like the following path:
> abfss://storagefs@mystorageaccount.dfs.core.windows.net/cod-ouiftSmthLikeThis/hbase/.hbase-snapshot

This confusion was caused by the default backup location set to the logs container during the environment creation. 


Follow-up questions: 
1. Where is an official documentation or guidelines location addressing this topic?
2. What backups are stored in the default location? Only the cluster backups? 
3. How can the Hadoop configuration be modified to allow the use of a different container for storing manual backups?

4. How can we give HBase the rights to access different storage locations?