Atlas Metadata server not starting

[#1]

Hello Team,

After enabling Kerberos on HDP-2.6 cluster, Atlas metadata server is not starting. Getting following error:

java exception
ERROR Java::OrgApacheHadoopHbaseIpc::RemoteWithExtrasException: org.apache.hadoop.hbase.coprocessor.CoprocessorException: HTTP 503 Error: HTTP 503
    at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.grant(RangerAuthorizationCoprocessor.java:1236)
    at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService$1.grant(AccessControlProtos.java:9933)
    at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService.callMethod(AccessControlProtos.java:10097)
    at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7857)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1999)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1981)
    at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32389)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)

How to solve this? Please suggest.

Thanks,

Bhushan

[#2]

@Bhushan Kandalkar

This looks like an authorization issue in ranger, Atlas need the hbase DB for the backend ensure all hbase components are running, Check the in Ranger that Atlas has the correct permissions to 2 important tables Atlas_titan and ATLAS_ENTITY_AUDIT_EVENTS see attached screenshots

Log on the Zookeeper CLI and check the ACL's you should have a similar output

[zk: localhost:2181(CONNECTED) 0] getAcl  /hbase-unsecure/table/ATLAS_ENTITY_AUDIT_EVENTS
'world,'anyone
: cdrwa
[zk: localhost:2181(CONNECTED) 1] getAcl  /hbase-unsecure/table/atlas_titan
'world,'anyone
: cdrwa
[zk: localhost:2181(CONNECTED)2] getAcl  /hbase-unsecure/table/hbase:meta
'world,'anyone
: cdrwa
[zk: localhost:2181(CONNECTED) 3] getAcl  /hbase-unsecure/table/hbase:namespace
'world,'anyone
: cdrwa

Please revert

[#3 · Accepted solution]

This issue occurred as we have used Load balancer while implementing Ranger HA. Breaking HA and keeping only one Ranger Admin resolved this issue for us.

[#4]

@Bhushan Kandalkar

Good it worked out but you shouldn't have omitted the information about the architecture ie Load balancer such info is critical in the analysis ....:-)

Happy hadooping