Atlas Metadata server start fail with hbase table grant operation denied by ranger

[#1]

Hi ,Atlas Metadata server start fail and i find the reason is the hbase table grant operation was denied by ranger. The doc has said that the permissions do not have the grant. I don't know why.

the audit log and ranger policy:

here is the log:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py", line 231, in <module>
    MetadataServer().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 720, in restart
    self.start(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py", line 92, in start
    user=params.hbase_user
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 273, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 293, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-venus_bigdata@VENUS.COM; cat /var/lib/ambari-agent/tmp/atlas_hbase_setup.rb | hbase shell -n' returned 1. atlas_titan
ATLAS_ENTITY_AUDIT_EVENTS
atlas
TABLE
ATLAS_ENTITY_AUDIT_EVENTS
access_tracker
alertDataSource
alertExecutor
alertStream
alertStreamSchema
alertdef
alertdetail
atlas_titan
eagle_metric
eaglehdfs_alert
enrichment
fileSensitivity
hiveResourceSensitivity
ipzone
mlmodel
pcap
pcapfiles
streamMetadata
streamdef
t
threatintel
userprofile
23 row(s) in 0.3190 seconds

nil
TABLE
ATLAS_ENTITY_AUDIT_EVENTS
access_tracker
alertDataSource
alertExecutor
alertStream
alertStreamSchema
alertdef
alertdetail
atlas_titan
eagle_metric
eaglehdfs_alert
enrichment
fileSensitivity
hiveResourceSensitivity
ipzone
mlmodel
pcap
pcapfiles
streamMetadata
streamdef
t
threatintel
userprofile
23 row(s) in 0.0170 seconds

nil
java exception
ERROR Java::OrgApacheHadoopHbaseIpc::RemoteWithExtrasException: org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.security.AccessControlException: Permission denied.
 at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.grant(RangerAuthorizationCoprocessor.java:1168)
 at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService$1.grant(AccessControlProtos.java:9933)
 at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService.callMethod(AccessControlProtos.java:10097)
 at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7717)
 at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1897)
 at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1879)
 at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32299)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2127)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
 at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
 at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
 at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.security.AccessControlException: Permission denied.
 at org.apache.ranger.admin.client.RangerAdminRESTClient.grantAccess(RangerAdminRESTClient.java:168)
 at org.apache.ranger.plugin.service.RangerBasePlugin.grantAccess(RangerBasePlugin.java:308)
 at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.grant(RangerAuthorizationCoprocessor.java:1161)
 ... 11 more

[#2]

can you please check if hbase policy refresh is happening properly

[#3]

How to check? i‘ve restart ranger many times, and not work.

[#4]

there are following ways to check :

1) go to ranger audit page, click on plugin tab and see if there is entry for hbase , and check for update time it should be greater than the policy update time in admin audit log

2) or you can go to the hbase logs and check there should be no failure log for policy updation

3) check access audit log in /var/log/ranger/admin/ , there will be lot of logging reg, call sent to ranger admin and their response too, see if the response is 200/304 , if it is then policy refresh is working fine.

[#5]

thanks @Deepak Sharma. i find the problem in the log

the hbase log:

2017-02-28 15:48:58,330 ERROR [Thread-74] client.RangerAdminRESTClient: Error getting policies. secureMode=true, user=hbase/bigdata6@VENUS.COM (auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0}, serviceName=venus_bigdata_hbase 2017-02-28 15:48:58,330 ERROR [Thread-74] util.PolicyRefresher: PolicyRefresher(serviceName=venus_bigdata_hbase): failed to refresh policies. Will continue to use last known version of policies (4) java.lang.Exception: HTTP 401 at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:126) at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:232) at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:188) at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:158)

and the ranger access log:

192.168.55.205 - - [28/Feb/2017:15:18:24 +0800] "GET /service/plugins/secure/policies/download/venus_bigdata_hbase?lastKnownVersion=4&pluginId=hbaseMaster@bigdata6-venus_bigdata_hbase HTTP/1.1" 401 - 192.168.55.206 - - [28/Feb/2017:15:18:24 +0800] "GET /service/plugins/secure/policies/download/venus_bigdata_hbase?lastKnownVersion=4&pluginId=hbaseRegional@bigdata7-venus_bigdata_hbase HTTP/1.1" 401 - 192.168.55.207 - - [28/Feb/2017:15:18:25 +0800] "GET /service/plugins/secure/policies/download/venus_bigdata_hbase?lastKnownVersion=4&pluginId=hbaseRegional@bigdata8-venus_bigdata_hbase HTTP/1.1" 401

------------

the ranger audit page have no edit history recently,and the same to other service. i don't know how to solve it. Please give me some advice, thanks.

[#6]

when i send the request use curl -i --negotiate -u hbase "http://bigdata6:6080/service/plugins/secure/policies/download/venus_bigdata_hbase?lastKnownVersion=4&pluginId=hbaseMaster@bigdata6-venus_bigdata_hbase", i return the results, i don't know the way hbase plugin works.

[#7]

li zhen can you please confirm following:

1)which hdp version are you using ?

2) and is this a secure cluster?

and if it is secure cluster then can you please check hbase repo ( bigdata6-venus_bigdata_hbase) on ranger UI, whether they are having following properties:

policy.download.auth.users

policy.grantrevoke.auth.users

tag.download.auth.users

if not please add these properties and add hbase user in its value please see the screenshot.:

screen-shot-2017-02-28-at-31040-pm.png

[#8]

i use hdp 2.5.3.0 and the cluster has kerberised. And the properties you mentioned have added, but not work.

[#9]

can you please see the screenshot and did you add the same properties, make sure you add the hbase user, if you have any custom hbase user then add that in these properites instead of hbase. and make sure you add it into your corresponding hbase repo venus_bigdata_hbase. after doing also if you see issue then just post the error if it is something different.

[#10]

thanks，i am sure i have add the same properties and use the hbase user.

but i find other service policy update also not work.

here is the ranger access log:

192.168.55.205 - - [01/Mar/2017:10:00:03 +0800] "GET /service/plugins/secure/policies/download/venus_bigdata_hbase?lastKnownVersion=4&pluginId=hbaseMaster@bigdata6-venus_bigdata_hbase HTTP/1.1" 401 - 192.168.55.205 - - [01/Mar/2017:10:00:03 +0800] "GET /service/plugins/secure/policies/download/venus_bigdata_hive?lastKnownVersion=3&pluginId=hiveServer2@bigdata6-venus_bigdata_hive HTTP/1.1" 401 - 192.168.55.206 - - [01/Mar/2017:10:00:04 +0800] "GET /service/plugins/secure/policies/download/venus_bigdata_hbase?lastKnownVersion=4&pluginId=hbaseRegional@bigdata7-venus_bigdata_hbase HTTP/1.1" 401 - 192.168.55.207 - - [01/Mar/2017:10:00:04 +0800] "GET /service/plugins/policies/download/venus_bigdata_storm?lastKnownVersion=3&pluginId=storm@bigdata8-venus_bigdata_storm HTTP/1.1" 304 - 192.168.55.205 - - [01/Mar/2017:10:00:05 +0800] "GET /service/plugins/secure/policies/download/venus_bigdata_hadoop?lastKnownVersion=2&pluginId=hdfs@bigdata6-venus_bigdata_hadoop HTTP/1.1" 401 - 192.168.55.206 - - [01/Mar/2017:10:00:05 +0800] "GET /service/plugins/secure/policies/download/venus_bigdata_hadoop?lastKnownVersion=2&pluginId=hdfs@bigdata7-venus_bigdata_hadoop HTTP/1.1" 401 - 192.168.55.207 - - [01/Mar/2017:10:00:05 +0800] "GET /service/plugins/secure/policies/download/venus_bigdata_hbase?lastKnownVersion=4&pluginId=hbaseRegional@bigdata8-venus_bigdata_hbase HTTP/1.1" 401 - 192.168.55.205 - - [01/Mar/2017:10:00:05 +0800] "GET /service/plugins/secure/policies/download/venus_bigdata_yarn?lastKnownVersion=2&pluginId=yarn@bigdata6-venus_bigdata_yarn HTTP/1.1" 401 - 192.168.55.208 - - [01/Mar/2017:10:00:07 +0800] "GET /service/plugins/secure/policies/download/venus_bigdata_hbase?lastKnownVersion=4&pluginId=hbaseRegional@bigdata9-venus_bigdata_hbase HTTP/1.1" 401 - 192.168.55.205 - - [01/Mar/2017:10:00:08 +0800] "GET /login.jsp HTTP/1.1" 200 3325 192.168.55.207 - - [01/Mar/2017:10:00:11 +0800] "GET /service/plugins/policies/download/venus_bigdata_storm?lastKnownVersion=3&pluginId=storm@bigdata8-venus_bigdata_storm HTTP/1.1" 304 - 192.168.55.208 - - [01/Mar/2017:10:00:13 +0800] "GET /service/plugins/secure/policies/download/venus_bigdata_kafka?lastKnownVersion=4&pluginId=kafka@bigdata9-venus_bigdata_kafka HTTP/1.1" 401 - 192.168.55.206 - - [01/Mar/2017:10:00:13 +0800] "GET /service/plugins/secure/policies/download/venus_bigdata_kafka?lastKnownVersion=4&pluginId=kafka@bigdata7-venus_bigdata_kafka HTTP/1.1" 401 - 192.168.55.208 - - [01/Mar/2017:10:00:13 +0800] "GET /service/plugins/secure/policies/download/venus_bigdata_kafka?lastKnownVersion=4&pluginId=kafka@bigdata9-venus_bigdata_kafka HTTP/1.1" 304 - 192.168.55.206 - - [01/Mar/2017:10:00:13 +0800] "GET /service/plugins/secure/policies/download/venus_bigdata_kafka?lastKnownVersion=4&pluginId=kafka@bigdata7-venus_bigdata_kafka HTTP/1.1" 304 - 192.168.55.205 - - [01/Mar/2017:10:00:33 +0800] "GET /service/plugins/secure/policies/download/venus_bigdata_hbase?lastKnownVersion=4&pluginId=hbaseMaster@bigdata6-venus_bigdata_hbase HTTP/1.1" 401 - 192.168.55.205 - - [01/Mar/2017:10:00:33 +0800] "GET /service/plugins/secure/policies/download/venus_bigdata_hive?lastKnownVersion=3&pluginId=hiveServer2@bigdata6-venus_bigdata_hive HTTP/1.1" 401 -