Atlas Metadata server start fail with hbase table grant operation denied by ranger

[#1]

Hi ,Atlas Metadata server start fail and i find the reason is the hbase table grant operation was denied by ranger. The doc has said that the permissions do not have the grant. I don't know why.

the audit log and ranger policy:

here is the log:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py", line 231, in <module>
    MetadataServer().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 720, in restart
    self.start(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py", line 92, in start
    user=params.hbase_user
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 273, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 293, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-venus_bigdata@VENUS.COM; cat /var/lib/ambari-agent/tmp/atlas_hbase_setup.rb | hbase shell -n' returned 1. atlas_titan
ATLAS_ENTITY_AUDIT_EVENTS
atlas
TABLE
ATLAS_ENTITY_AUDIT_EVENTS
access_tracker
alertDataSource
alertExecutor
alertStream
alertStreamSchema
alertdef
alertdetail
atlas_titan
eagle_metric
eaglehdfs_alert
enrichment
fileSensitivity
hiveResourceSensitivity
ipzone
mlmodel
pcap
pcapfiles
streamMetadata
streamdef
t
threatintel
userprofile
23 row(s) in 0.3190 seconds

nil
TABLE
ATLAS_ENTITY_AUDIT_EVENTS
access_tracker
alertDataSource
alertExecutor
alertStream
alertStreamSchema
alertdef
alertdetail
atlas_titan
eagle_metric
eaglehdfs_alert
enrichment
fileSensitivity
hiveResourceSensitivity
ipzone
mlmodel
pcap
pcapfiles
streamMetadata
streamdef
t
threatintel
userprofile
23 row(s) in 0.0170 seconds

nil
java exception
ERROR Java::OrgApacheHadoopHbaseIpc::RemoteWithExtrasException: org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.security.AccessControlException: Permission denied.
 at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.grant(RangerAuthorizationCoprocessor.java:1168)
 at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService$1.grant(AccessControlProtos.java:9933)
 at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService.callMethod(AccessControlProtos.java:10097)
 at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7717)
 at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1897)
 at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1879)
 at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32299)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2127)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
 at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
 at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
 at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.security.AccessControlException: Permission denied.
 at org.apache.ranger.admin.client.RangerAdminRESTClient.grantAccess(RangerAdminRESTClient.java:168)
 at org.apache.ranger.plugin.service.RangerBasePlugin.grantAccess(RangerBasePlugin.java:308)
 at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.grant(RangerAuthorizationCoprocessor.java:1161)
 ... 11 more

[#2]

can you please do kinit with hbase keytab and try the call to download the policies /service/plugins/secure/policies/download/venus_bigdata_hbase?lastKnownVersion=4&pluginId=hbaseRegional@bigdata9-venus_bigdata_hbase , just to confirm if everything is fine with keytab?

do check one more thing if core-site.xml is present in /etc/ranger/admin/conf , if not just copy it there and restart the ranger admin

[#3]

@li zhen

Few questions

1- Is this a HA cluster? If so, is Ranger configured to talk to LoadBalancer. If Loadbalancer is configured then there is an additional step that should be done as defined in this docs.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-upgrade/content/upgrading_HDP_pos...

2- is this cluster initially simple and later kerberos is enabled?

3- Also for any download request, you will notice 401 followed by 200(if there are changes to policies in repo) or 401 followed by 304(when there is no change in policies for this repo)

[#4]

In case someone faces the same issue: in my case, I solved it by ensuring that 'atlas' user is known in Ranger.

[#5]

Pierre solution is correct.

If you installed Atlas after Ranger UserSync has been configured to use LDAP, new local users will not get synced in ranger like atlas. This user is needed to setup hbase tables.

To fix, revert UserSync to UNIX, restart only Ranger UserSync, Switch back to UserSync LDAP config. In Ranger add user atlas to HBase all policy. Restart Atlas.