Created on 02-24-2017 09:51 AM - edited 08-19-2019 03:12 AM
Hi ,Atlas Metadata server start fail and i find the reason is the hbase table grant operation was denied by ranger. The doc has said that the permissions do not have the grant. I don't know why.
the audit log and ranger policy:
here is the log:
Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py", line 231, in <module> MetadataServer().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute method(env) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 720, in restart self.start(env, upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py", line 92, in start user=params.hbase_user File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 273, in action_run tries=self.resource.tries, try_sleep=self.resource.try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 293, in _call raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of 'kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-venus_bigdata@VENUS.COM; cat /var/lib/ambari-agent/tmp/atlas_hbase_setup.rb | hbase shell -n' returned 1. atlas_titan ATLAS_ENTITY_AUDIT_EVENTS atlas TABLE ATLAS_ENTITY_AUDIT_EVENTS access_tracker alertDataSource alertExecutor alertStream alertStreamSchema alertdef alertdetail atlas_titan eagle_metric eaglehdfs_alert enrichment fileSensitivity hiveResourceSensitivity ipzone mlmodel pcap pcapfiles streamMetadata streamdef t threatintel userprofile 23 row(s) in 0.3190 seconds nil TABLE ATLAS_ENTITY_AUDIT_EVENTS access_tracker alertDataSource alertExecutor alertStream alertStreamSchema alertdef alertdetail atlas_titan eagle_metric eaglehdfs_alert enrichment fileSensitivity hiveResourceSensitivity ipzone mlmodel pcap pcapfiles streamMetadata streamdef t threatintel userprofile 23 row(s) in 0.0170 seconds nil java exception ERROR Java::OrgApacheHadoopHbaseIpc::RemoteWithExtrasException: org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.security.AccessControlException: Permission denied. at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.grant(RangerAuthorizationCoprocessor.java:1168) at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService$1.grant(AccessControlProtos.java:9933) at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService.callMethod(AccessControlProtos.java:10097) at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7717) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1897) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1879) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32299) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2127) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.security.AccessControlException: Permission denied. at org.apache.ranger.admin.client.RangerAdminRESTClient.grantAccess(RangerAdminRESTClient.java:168) at org.apache.ranger.plugin.service.RangerBasePlugin.grantAccess(RangerBasePlugin.java:308) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.grant(RangerAuthorizationCoprocessor.java:1161) ... 11 more
Created 03-01-2017 03:54 AM
can you please do kinit with hbase keytab and try the call to download the policies /service/plugins/secure/policies/download/venus_bigdata_hbase?lastKnownVersion=4&pluginId=hbaseRegional@bigdata9-venus_bigdata_hbase , just to confirm if everything is fine with keytab?
do check one more thing if core-site.xml is present in /etc/ranger/admin/conf , if not just copy it there and restart the ranger admin
Created 03-29-2017 11:41 PM
1- Is this a HA cluster? If so, is Ranger configured to talk to LoadBalancer. If Loadbalancer is configured then there is an additional step that should be done as defined in this docs.
2- is this cluster initially simple and later kerberos is enabled?
3- Also for any download request, you will notice 401 followed by 200(if there are changes to policies in repo) or 401 followed by 304(when there is no change in policies for this repo)
Created 10-30-2017 11:01 AM
In case someone faces the same issue: in my case, I solved it by ensuring that 'atlas' user is known in Ranger.
Created 04-05-2018 12:27 AM
Pierre solution is correct.
If you installed Atlas after Ranger UserSync has been configured to use LDAP, new local users will not get synced in ranger like atlas. This user is needed to setup hbase tables.
To fix, revert UserSync to UNIX, restart only Ranger UserSync, Switch back to UserSync LDAP config. In Ranger add user atlas to HBase all policy. Restart Atlas.